Commons:Spacemedia

From Wikimedia Commons, the free media repository
Jump to navigation Jump to search

Spacemedia is a tool written by Don-vip that continuously harvests media libraries of various space agencies in order to find free media not yet uploaded to Wikimedia Commons.

The tool runs on Toolforge (Kubernetes/jdk17 cluster) and uses OptimusPrimeBot account to upload automatically new discovered media.

The discussion to authorize this tool takes place here.

As of 2024-08-10 the tool has already imported more than 250.000 files.

Since August 2024 the scope of this project has been expanded to include non-space-related pictures (such as Olympics 2024 or U.S. Government pictures).

Links

[edit]

Useful discussions / projects

[edit]

Monitored repositories and status

[edit]

Spacemedia sources

[edit]

Spacemedia_files_(review_needed)

Repositories providing free media with compatible licences
Agency / repository Licence(s) Remark Status
US Air Force / Space Force
https://www.flickr.com/photos/airforcespacecommand Cc-by-2.0, Cc-zero Should be PD-USGov-Military-Air Force ✓ Done
https://www.flickr.com/people/129133022@N07/ PD Should be PD-USGov-Military-Air Force ✓ Done
https://www.dvidshub.net/search/?q=&filter%5Btype%5D=image&filter%5Bbranch%5D=Space+Force&view=grid&sort=publishdate PD-USGov-Military-Space Force ✓ Done
https://www.spaceforce.mil/Multimedia/Photos/ PD-USGov-Military-Space Force ✓ Done
DLR (Deutsches Zentrum für Luft- und Raumfahrt, German Aerospace Center)
https://www.flickr.com/photos/dlr_de Cc-by-2.0, Cc-by-sa-2.0 ✓ Done
https://www.dlr.de/EN/organisation-dlr/news/all-news.html Cc-by-sa-3.0 Many news images in cc-by-sa, need to check individually though  Not done
ESA (European Space Agency)
http://www.esa.int/spaceinimages/Images Cc-by-sa-3.0-IGO ✓ Done
https://www.esa.int/esatv/content/search?SearchText=CC-BY Cc-by-sa-3.0-IGO  Not done
https://www.flickr.com/photos/europeanspaceagency/ Cc-by-2.0, Cc-by-sa-2.0, CC-PD-Mark ✓ Done
https://www.flickr.com/photos/esa_events Cc-by-2.0, Cc-by-sa-2.0, CC-PD-Mark ✓ Done
https://www.youtube.com/channel/UClB8L8TJEQfZ41Ii0gJRTSQ Cc-by-sa-3.0-IGO  Not done
EU (European Union)
https://emergency.copernicus.eu/mapping/list-of-activations-rapid - https://emergency.copernicus.eu/mapping/list-of-activations-risk-and-recovery Attribution-Copernicus required to inform about the source of information and/or data by using provided reference  Not done
https://www.copernicus.eu/en/media/image-day Attribution-Copernicus ✓ Done
https://erccportal.jrc.ec.europa.eu/ECHO-Products/Maps#/maps/latest PD-European-Commission Category:ECHO Daily Map of 2023 ✓ Done
Observatories, Space telescopes, International Astronomical Union
https://www.eso.org/public/images/ ESO (CC-by-4.0)  Half done
https://www.iau.org/public/images/ CC-by-4.0  Half done
https://www.spacetelescope.org/images/ ESA-Hubble  Half done
https://hubblesite.org/resource-gallery/images PD-Hubble  Half done
https://webbtelescope.org/ PD-Webb ✓ Done
https://esawebb.org/ ESA-Webb ✓ Done
https://gallery.lsst.org Cc-by-4.0 Webdam (old)  Not done
https://rubin.canto.com/v/gallery/ Cc-by-4.0 Canto (new)  Not done
https://www.almaobservatory.org/en/copyright-notice/ Cc-by-4.0  Not done
https://noirlab.edu/public/images/ NOIRLab ✓ Done
https://nso.edu/about/image-use-policy/ Cc-By 4.0  Not done
INPE (Instituto Nacional de Pesquisas Espaciais) - Brazil
http://www.dpi.inpe.br/galeria/ INPE, Cc-By-Sa 4.0 ✓ Done
https://www.flickr.com/photos/observacao-da-terra/ INPE, Cc-By-Sa 2.0 ✓ Done
ISRO (Indian Space Research Organisation)
http://pib.gov.in/PhotoCategories.aspx?MenuId=8 GODL-India (through PIB) https://gitlab.com/Gazoth/pib-upload/issues/1  Not done
KARI (Korean Aerospace Research Institute)
https://www.kari.re.kr/kor/kariimg/list.do?img_gbn=PHO KOGL ✓ Done
NASA (National Aeronautics and Space Administration) - live
https://images.nasa.gov/ PD-USGov-NASA  Half done
https://www.nasa.gov/image-article/ PD-USGov-NASA  Half done
https://www.nasa.gov/gallery/ PD-USGov-NASA  Half done
https://photojournal.jpl.nasa.gov/ PD-USGov-NASA Use Template:NASA Photojournal  Half done
https://earthobservatory.nasa.gov/images PD-USGov-NASA Use Template:ISS Crew Earth Observations  Not done
https://eol.jsc.nasa.gov/Collections/ PD-USGov-NASA To coordinate with User_talk:Askeuhd/Archive_2#Usefulness_of_ISS_image_dumps

Use Template:ISS Crew Earth Observations

 Not done
https://www.nasa.gov/multimedia/imagegallery/iotd.html PD-USGov-NASA To do. https://kaijento.github.io/2017/05/06/web-scraping-nasa-image-of-the-day/  Not done
https://solarsystem.nasa.gov/resources/ PD-USGov-NASA  Not done
https://www.sti.nasa.gov/harvesting-data-from-ntrs/ PD-USGov-NASA To do. Run requests between 8PM-8AM U.S. ET. No more than one request every 3 seconds.  Not done
https://svs.gsfc.nasa.gov/ PD-USGov-NASA  Half done
https://www.uahirise.org/ PD-NASA-HiRISE https://www.uahirise.org/media/usage.php  Not done
https://visibleearth.nasa.gov/ PD-USGov-NASA  Not done
https://asterweb.jpl.nasa.gov/ PD-USGov-NASA ✓ Done
https://modis.gsfc.nasa.gov/gallery/showall.php PD-USGov-NASA  Half done
https://sdo.gsfc.nasa.gov/ PD-USGov-NASA ✓ Done
https://epic.gsfc.nasa.gov/galleries PD-USGov-NASA  Not done
https://ails.arc.nasa.gov/ PD-USGov-NASA  Not done
https://chandra.si.edu/photo/ PD-USGov-NASA  Half done
https://aria.jpl.nasa.gov/ PD-USGov-NASA  Not done
https://www.dfrc.nasa.gov/Gallery/Photo/index.html PD-USGov-NASA  Not done
https://www.flickr.com/photos/atmospheric-infrared-sounder Cc-by-2.0 Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/earthrightnow Cc-by-2.0 Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/gsfc Cc-by-2.0, Cc-zero Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/morpheuslander Cc-by-2.0 Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasa2explore Cc-by-2.0, CC-PD-Mark Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasa_appel Cc-by-2.0 Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasa_goddard Cc-by-2.0 Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasa_ice Cc-by-2.0 Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasablueshift Cc-by-2.0, Cc-by-sa-2.0 Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasacommons Flickr-no known copyright restrictions Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasaearthobservatory Cc-by-2.0 Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasafo CC-PD-Mark Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasahubble Cc-by-2.0, CC-PD-Mark Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasakennedy Cc-by-2.0, Cc-by-sa-2.0, CC-PD-Mark Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasarobonaut Cc-by-2.0 Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/nasawebbtelescope Cc-by-2.0 Should be PD-USGov-NASA  Half done
https://www.flickr.com/photos/uahirise-mars Cc-zero, CC-PD-Mark Should be PD-USGov-NASA  Half done
https://video.ibm.com/channel/nasa-tv-wallops TBC Should be PD-USGov-NASA  Not done
https://nasa-external-ocomm.app.box.com/s/onrtmdvofqluv5ei5kfu5u1pf8v4xqtl/folder/109925769802 PD-USGov-NASA ✓ Done
https://www.missionjuno.swri.edu/junocam/processing?source=public&ob_from=&ob_to=&perpage=100 Cc-By Be careful, the licence must be checked for each image, some are cc-by-nc  Not done
https://lroc.sese.asu.edu/posts PD-USGov-NASA ✓ Done
https://www.shadowcam.asu.edu/images PD-USGov-NASA ✓ Done
https://tothemoon.ser.asu.edu/ PD-USGov-NASA  Not done
NASA (National Aeronautics and Space Administration) - dead
http://grin.hq.nasa.gov PD-USGov-NASA Category:Great Images in NASA - Commons:Batch uploading/Great Images in NASA ✓ Done
https://www.ssc.nasa.gov/sirs/scripts/xmlWelcome.pl PD-USGov-NASA Stennis Image Retrieval System (SIRS) - shut down around 2020  Not done
http://lisar.larc.nasa.gov/UTILS/info.cgi? PD-USGov-NASA Langley Image Scanning, Archival and Retrieval  Not done
https://mix.msfc.nasa.gov/ PD-USGov-NASA Marshall image eXchange  Not done
NOAA (National Oceanic and Atmospheric Administration)
https://www.flickr.com/photos/noaasatellites/ CC-PD-Mark Should be PD-USGov-NOAA or PD-USGov-NASA ✓ Done
https://www.noaa.gov/digital-library/collections/2268 PD-USGov-NOAA ✓ Done
https://www.nesdis.noaa.gov/real-time-imagery/imagery-collections PD-USGov-NOAA ✓ Done
https://www.ngdc.noaa.gov/stp/satellite/goes-r.html PD-USGov-NOAA Curated SUVI images and animations processed at the National Centers for Environmental Information (NCEI)  Not done
http://sxi.ngdc.noaa.gov/ PD-USGov-NOAA Older SXI image data from GOES 12-15  Not done
https://www.ncei.noaa.gov/data/goes-r-series-l2-operational-space-weather-products/ PD-USGov-NOAA Original operational versions of SUVI images are also available through NCEI  Not done
https://repository.library.noaa.gov/cbrowse?pid=noaa%3A9&parentId=noaa%3A9 Many  Not done
https://satlib.cira.colostate.edu/ PD-USGov-NOAA  Not done
USGS (United States Geological Survey)
https://lima.usgs.gov/access.php PD-USGov-USGS  Not done
https://astrogeology.usgs.gov/site/annex PD-USGov-USGS + PD-USGov-NASA  Not done
Individuals
https://www.flickr.com/photos/charlesandmaggie/ Cc-by-2.0 ✓ Done
https://www.flickr.com/photos/geckzilla Cc-by-2.0 ✓ Done
https://www.flickr.com/photos/harrystrangerphotographyhttps://www.flickr.com/photos/194849271@N04 PD ✓ Done
https://www.flickr.com/photos/192271236@N03 Cc-by-2.0 ✓ Done
https://www.flickr.com/people/kevinmgill Cc-by-2.0 ✓ Done
https://www.flickr.com/photos/pierre_markuse Cc-by-2.0 ✓ Done
Universities
https://www.cfa.harvard.edu/resources/image-gallery PD https://www.cfa.harvard.edu/news/image-use-policy  Not done
https://www.cassis.unibe.ch/ Cc-By-Sa 3.0 IGO  Not done
https://cimss.ssec.wisc.edu/satellite-blog/ PD https://cimss.ssec.wisc.edu/satellite-blog/about  Not done
Other public institutions
https://openspot-dinamis.data-terra.org/ Licence Ouverte 2  Half done
Private companies
https://www.youtube.com/@arianespace Videos no longer free :( ✓ Done
https://www.planet.com/gallery/ Formerly Cc-by-4.0. No longer :( ✓ Done
https://www.flickr.com/photos/sentinelhub Cc-By 2.0 ✓ Done
https://www.flickr.com/photos/spacex Formerly Cc-zero-SpaceX. No longer :( ✓ Done
https://registry.opendata.aws/umbra-open-data/ Cc-By 4.0 https://umbra.space/open-license ✓ Done
https://registry.opendata.aws/capella_opendata/ Cc-By 4.0 ✓ Done

Diplomedia side-project sources

[edit]

Diplomedia_files_(review_needed)

Repositories providing free media with compatible licences
Department / repository Licence(s) Remark Status
US Government
Hundreds of U.S. Embassies and Consulates Flickr accounts PD-USGov-DOS  Half done

Govmedia side-project sources

[edit]

Govmedia_files_(review_needed)

Repositories providing free media with compatible licences
Department / repository Licence(s) Remark Status
US Government
https://www.flickr.com/photos/whitehouse PD-USGov-POTUS ✓ Done
https://www.flickr.com/photos/whitehouse45 PD-USGov-POTUS ✓ Done
https://www.flickr.com/photos/obamawhitehouse PD-USGov-POTUS ✓ Done
https://www.flickr.com/photos/statephotos PD-USGov-DOS  Half done

Milimedia side-project sources

[edit]

Milimedia_files_(review_needed)

Repositories providing free media with compatible licences
Branch / Unit / repository Licence(s) Remark Status
US Military
https://www.dvidshub.net PD-USGov-Military  Half done

Scientimedia side-project sources

[edit]

Scientimedia_files_(review_needed)

Repositories providing free media with compatible licences
Organization / repository Licence(s) Remark Status
French organizations
https://image.ifremer.fr/ CC-BY Océanothèque  Half done
German organizations
https://media.marum.de/index.php?/category/96 CC-BY Marum Media Library  Not done
https://multimedia.awi.de/ CC-BY Alfred Wegener Institute  Not done
Swiss organizations
https://cds.cern.ch/collection/Creative%20Commons%20Images%20from%20CERN?ln=fr CC-BY CERN  Not done

Technical description

[edit]

The tool is developed using Maven, Spring Boot, Spring Data JPA, Spring Cache, Jsoup, Dozer, Hibernate Search, Flickr4Java and Apache Commons libraries.

The tool connects to the Commons database replica to search for existing medias (by their base36 SHA-1) in these three tables:

  • image: current version of images
  • oldimage: old versions of images
  • filearchive: deleted images (when another version with a different SHA-1 exists)

The result of the analysis is persisted in a dedicated user database.

Most of the configuration is in application.properties. Sensitive information are not stored in the git repository and must be provided at runtime, on the command line.

Administration

[edit]

Commons:Spacemedia/Administration

Future

[edit]