I'm diving into data science and I'm looking for a complete database of minerals. I downloaded the csv file from http://rruff.info/ima/ . It contains 5516 lines of data but it does not have images or urls. Does anyone know if there is a database of minerals with images? I'd like to find it to train a neural network.
$\begingroup$
$\endgroup$
8
-
2$\begingroup$ Do you want a neural network to do image recognition, and based on how the images look in the database try and match an unknown image to a mineral? This will not work. You are more than welcome to ask a new question and I will explain why. $\endgroup$– GimelistNov 19, 2019 at 10:07
-
$\begingroup$ If that's your purpose, as @Gimelist says this can't be done. If not Google would have done it. But it would be nice to read the expert explanation why not. My friends told me that "make an app that recognizs minerals". I doubt this is possible was my answer. But maybe you can clarify what you wanna do. Maybe with AI you can do some other interesting things $\endgroup$– user18261Nov 19, 2019 at 12:13
-
$\begingroup$ Gimesist, yes it is for image recognition. I'm pretty confident it will work for common minerals with distinctive visual characteristics. I just need a good dataset and pictures from multiple angles. $\endgroup$– Pedro AguiarNov 19, 2019 at 22:40
-
$\begingroup$ @PedroAguiar visual shape and colour are two of the most misleading properties of minerals used for identification. This was one of the reasons why we banned mineral ID questions - because we had a photo and this was mostly useless. Indeed, some minerals are so distinctive (sometimes) that this might work. But I can count 10 minerals I can identify only by looking at them (out of thousands), and only when they are in that specific form. $\endgroup$– GimelistNov 19, 2019 at 23:43
-
2$\begingroup$ @PedroAguiar over 10 years experience of mineral ID has taught me that it has hard to trust a visual mineral identification. Unless your neural network also takes into account hardness, lustre, density, chemical composition, geological context, it will probably not work. $\endgroup$– GimelistNov 19, 2019 at 23:45
|
Show 3 more comments
2 Answers
$\begingroup$
$\endgroup$
2
First of all rruff.info is an excelent site done by chemists to take data quickly, and you can totally trust the source.
Second, for images, most of the listed +5000 species are not shown in nature, and you cannot have pictures of all of them. You will only find on the net pictures from common minerals on nature.
Third it depends if your work is gonna be published or not, as you would need permission or CC licensed images if so.
Copyrigthed database of pictures and data:
Mindat can share with you their data, and they are going soon to open them on CC format with an API, but the pictures they have belong to each author and many of them are copyrigthed (and gonna stay copyrigthed if they open data, you would need to ask permission individually for each picture).
CC license pictures:
I wondered how to obtain pictures from common minerals on CC license because I am writing a mineral app and my choise was Dr Lavinsky collection uploaded to wikipedia from his mineral shop iRocks.com.
At his website they have plentifull of high quality well described pictures of common minerals you can maybe download with a script making GET petitions, and you can ask them for permission, even for the pictures that are not uploaded to the wiki. I did it myself for my mineral guide and they just ask me to quote iRocks.com; they are very fair people. Aparently they are happy to share if you quote his shop.
At Wikipedia you can find on CC license >2000 pictures uploaded from mindat.org, and >5000 uploaded from iRocks.com
You have the complete list of minerals with picture on Wikimedia Commons where to start to collect the data. I don't know if you could somehow extract all the images url using GET petitions with code. Myself I only took the iRocks pictures because my guide covered only 180 minerals, the descriptions Dr Lavinsky provides are valuable (example), and then I only needed to quote one author (I ended by taking them directly from iRocks and link on my work to each piece on his shop, they got very happy about that, they answered my mail with "many thanks").
-
$\begingroup$ Thank you Leukocyte. I'm happy to know that they are going to open the database and create an API, my project will be open source too. I'm not going to publish any pictures they will be used to train a software to identify minerals then it can be used in classes where the students can use a smartphone to find out the name and the chemical composition of the mineral. About the amount of data, a list with the most common minerals should be enough to start. But to make it work the computer needs many pictures from the same mineral in many angles and light conditions. The real work is the DB. $\endgroup$ Nov 19, 2019 at 20:39
-
$\begingroup$ @Pedro Aguiar If it helps, here you can download the 180 minerals selection I did. I selected the minerals that I have seen on my geology degree. Anyhow, as Gimelist, I doubt you can find a way to identify minerals from a picture. A single specie can have three different habits. $\endgroup$– user18261Nov 20, 2019 at 11:32
$\begingroup$
$\endgroup$
6
I think www.mindat.org has what you are looking for but I don't know how easy it is to perform you own queries of the underlying database.
-
$\begingroup$ The problem for mindat is their database is copyrigthed. $\endgroup$– user18261Nov 19, 2019 at 6:03
-
$\begingroup$ Copyright doesn't necessarily mean they won't let you work with it. So it may still be worth the OP contacting them. $\endgroup$– haresfurNov 19, 2019 at 7:23
-
$\begingroup$ @Leukocyte mindat allow access to their database if contacted and convinced. $\endgroup$– GimelistNov 19, 2019 at 10:10
-
$\begingroup$ They cannot. If you read my link on comments they are going to open data and create an API, but each photograph belongs to each author and gonna stay copyrigthed and not available at the API. Poster could only use the pictures that were shared as CC on Wikipedia, but for the rest he should ask for permission to each author. For that, in what concerns pictures, it is better to take iRocks ones. They uploaded 5000 images to Wkipedia (mindat uploaded 2000, see link on my answer) and you only need to quote one author. For data rruff.info is better suited, unless mindat has localities. $\endgroup$– user18261Nov 19, 2019 at 10:11
-
$\begingroup$ @Gimelist not for pictures because they belong to each author. For data they are going to open data, as said in my link on comments. $\endgroup$– user18261Nov 19, 2019 at 10:17