Current efforts to apply machine learning techniques to the automated recognition of microstructures of welded joints of structural steels will be presented. The machine learning approach is based on the Convolution Neural Network (CNN) and deep learning techniques, that have attracted a lot of attention for automated image identification and regression. This work is in progress, however promising results were obtained for the basic microstructures during the first tests. A critical implementation of the commonly agreed definitions used in classification of microstructures (Thewlis, 2004, IIW Doc. iX-1533-88) is being used to train the algorithm. The characteristic microstructures such as Primary Ferrite, Pearlite, Ferrite and Second Phases, Acicular Ferrite and Martensite are being used as a main category for classification. Surrounding and localized features of the microstructures can be analyzed by using different training techniques. In this context, the choice of traditional point counting vs. “image segementation” (defining microstructural regions in a fashion similar to painting a map) will also be discussed. The approach proposed, aims to open the possibility of automated quality control of microstructures and of a “big data” approach to the study of microstructures. Ultimately, the classification and quantification of numerous micrographs can be used to anticipate the mechanical properties from a micrograph.