Software requirements :<\/p>\n
- \n
- Dongle with TM AI license<\/li>\n<\/ul>\n
Version requirements:<\/p>\n
- \n
- TMflow version\uff1a 1.88.2200(If you are using the above version, you need to match the relevant version of TM AI+ Training Server )<\/li>\n
- TM AI+ Training Server 2.12.2200(If you are using the above version, you need to match the relevant version of TMflow)<\/li>\n<\/ul>\n
\nIPC inspection task description<\/h2>\n
Inspection Process :<\/strong><\/h3>\n
Use the display panel of the IPC for Pattern matching positioning, and move the robot to the inspection area to inspect the following items in order<\/p>\n
1.Is the filter ring installed on the power line<\/p>\n
2.Whether the cable is actually inserted into the slot<\/p>\n
3.Are the screw holes screws<\/p>\n
4.Is the cable alignment method correct<\/p>\n
\nShooting screen adjustment<\/b><\/h2>\n
Place the IPC in the working plane<\/b><\/h3>\n
Determine the working enviroment is fixed, the IPC placed on the work plane, adjust the shooting angle, and open the light source, you can replace the polarizer to avoid reflections that lead to unclear images.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/\u96fb\u63a7\u7bb1.png\")
<\/a><\/p>\n
\nMove the robot to select the shooting area<\/strong><\/h3>\n
Move the robot to the desired location, use the controller to control the movement of the robot, and observe whether the object to be detected is within the visible range through Vision.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/\u89c0\u5bdf\u62cd\u651d\u7bc4\u570d.png\")
<\/a><\/p>\n
\nShooting images and parameter adjustment<\/b><\/h3>\n
Determine the field of view to be detected, place a TM calibration plate, open the light source (it is recommended that the use of polarizers to avoid reflections leading to unclear shooting), adjust the parameters of the camera.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/\u6821\u6b63\u7248\u8abf\u6574\u53c3\u6578.png\")
<\/a><\/p>\n
\nHow to select the ROI and image collection\u00a0<\/strong><\/h2>\n
Image collection instructions<\/h3>\n
When performing AI Classification, the entire screen is not captured, but the features within the detection range must be selected.\u00a0 The following examples show how to select ROI and choose the images to be captured for the following detection contents.<\/p>\n
- \n
- Is equipped with filter ring\n
- \n
- It is “OK” to equip with filter ring.<\/li>\n
- “NG” without magnetic ring equipped.<\/li>\n<\/ul>\n<\/li>\n
- Is the plugin actually inserted\n
- \n
- There is indeed inserted as “OK”.<\/li>\n
- Not really inserted or not inserted as NG.<\/li>\n<\/ul>\n<\/li>\n
- Whether the screw hole has locking screw attached\n
- \n
- Locking screw for “OK”.<\/li>\n
- Not locked with screws for “NG”.<\/li>\n<\/ul>\n<\/li>\n
- Is the cable alignment method correct\n
- \n
- Cable on and correct is “OK”.<\/li>\n
- Cable not connected as “NG”.<\/li>\n
- Cable interleaved as “CROSS”.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
\nImage collection instructions\u00a0– Is equipped with filter ring<\/strong><\/h3>\n
Attach the filter ring first. Because the filter ring is attached to the power line, there will be a certain range of movement. When making a frame selection for the filter ring, the possibility of moving the filter ring must be considered as much as possible. Refer to the following figure<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/ferriteroi.png\")
<\/a><\/p>\n
\nWhen collecting OK situations, various possibilities of the movement of the filter ring must be considered, so you can try to move the filter ring more to create various possibilities of the appearance of the filter ring . To ensure that there is enough information, it is recommended to collect about 30 images. The demonstration is as follows.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/ferriteOK.png\")
<\/a><\/p>\n
\nWhen collecting NG situations, various possibilities of the power cord must be considered, so you can try to create various possibilities of the power cord. In order to ensure enough information, it is recommended to collect about 30 images, as shown below.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/ferriteNG.png\")
<\/a><\/p>\n
\nImage collection instructions\u00a0-Is the plugin actually inserted<\/strong><\/h3>\n
Since the position of the plug-in is fixed, the plug-in screen must occupy most of the ROI screen, so as to avoid being affected by the background factors and affecting the AI judgment.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/insertroi.png\")
<\/a><\/p>\n
\nWhen collecting OK situations, since the plug-in changes little, you can try to change the angle of the wire head. It is recommended to collect about 30 images, or change the ambient light source to adapt to various environmental changes. If you want to change the ambient light source, it is recommended to collect more images to increase the information for AI learning.
\n![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/insertok.png\")
<\/a><\/p>\n
\nWhen collecting NG cases, you need to collect more possibilities of incomplete insertion, to prevent AI from confusing the cases that are almost inserted and fully inserted. At the same time, you also need to consider the possibility of no insertion at all. It is recommended to collect about 30 images.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/insertng.png\")
<\/a><\/p>\n
\nImage collection instructions-whether there is a screw in the screw hole_top shot of the IPC<\/strong><\/h3>\n
When selecting the screw hole, the hole position must also occupy more than half of the ROI. Considering that there are various angles or lighting problems when shooting screws, the selection range should not be too narrow, so that it can cope with different situations.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/upscrewroi.png\")
<\/a><\/p>\n
\nSince the state of the screw locking the screw does not change much, and the background is mostly metal surface, we can create additional light sources to increase the information of various environmental changes.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/spotlight.png\")
<\/a><\/p>\n
\nThe following shows the situation of collecting OK images. It is recommended to collect about 30 images. You can see that each image is affected by different light and shadow.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/upscrewok.png\")
<\/a><\/p>\n
\nWhwn collecting NG images, you can also refer to the way of collecting OK images. It is also recommended to collect about 30 images and consider the influence of different light soutces.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/upscrewng.png\")
<\/a><\/p>\n
\nImage collection instructions – whether there is a screw in the screw hole_side shot of the IPC<\/strong><\/h3>\n
After taking the upper screw, we move the robot to the side of the IPC and check whether the side screws are locked. Since there are various changes in the side shot, you can also create extra light sources to simulate different situations.<\/p>\n
<\/a><\/p>\n
\nThe screw hole should also occupy more than half of the ROI range, and the screw hole should be as close to the center of the ROI as possible, to prevent reflection from causing AI information misjudgment.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/sidescrewroi.png\")
<\/a><\/p>\n
\nSimilarly, when collectng OK situations, you can consider different light sources. In order to make AI able to judge more screw, you can add the situation of shooting the upper control box to this situation for training. It is recommended to collect about 30 images, combined with the situation of the upper control box, to cope with various situations.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/top-shot-screw-OK.png\")
<\/a><\/p>\n
\nAnd when collecting NG situations, you can also add the situation of shooting the upper control box to this situation for training. It is recommended to collect about 30 images, combined with the situation of the upper control box, to cope with various situations.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/top-shot-screw-NG.png\")
<\/a><\/p>\n
\nImage collection instructions – whether the cable is connected correctly\u00a0<\/b><\/h3>\n
The purpose of this task is to confirm whether the cable is properly\u00a0 inserted and inserted in the correct position, and therer should be no wrong or unplugged situations. The entire cable should also occupy most of the ROI range, and try not to have redundant backgrounds that affect AI judgment. We divide the cable into three situations for image collection.<\/p>\n
- \n
- One-to-one correspondence, overlapping lines are also accepted, regarded ad OK<\/li>\n
- The cable head is properly inserted, but inserted in the wrong hole, regarded as CROSS<\/li>\n
- The cable head is not properly inserted, regarded as NG<\/li>\n<\/ul>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/cableroi.png\")
<\/a><\/p>\n
\nImage collection instructions – whether the cable is connected correctly_OK situation<\/b><\/p>\n
One-to-one correspondence, overlapping lines are also accepted, make sure the cable head is properly inserted, create some opening and closing situations, and create overlapping cable heads, collect about 30 images, the following are all regarded as OK situations.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/06\/cable.png\")
<\/a><\/p>\n
\nImage collection instructions – whether the cable is connected correctly_CROSS situation<\/b><\/p>\n
The cable head is properly inserted, but inserted in the wrong hole, collect the situations of two lines crossing, three line crossing, and four lines crossing, and consider all possible combinations as much as possible. Collect 2 to 3 images for each of the above situations, so a total of about 30 images are collected.
\n![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/06\/cable_1.png\")
<\/a><\/p>\n
\nImage collection instructions – whether the cable is connected correctly_NG situation<\/b><\/p>\n
The cable head is not properly inserted, collect the situations where there may be four unplugged, three unplugged, two unplugged, or even all unplugged, collect about 30 images.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/cableng.png\")
<\/a><\/p>\n
\nImage collection instructions – image storage method<\/strong><\/h3>\n
There are two main items for storing images (for detailed storage instructions, please refer to the manual)<\/p>\n
- \n
- Store ROI image in AI Classification\n
- \n
- TM AI+ Training Server Dataset<\/li>\n
- TM SSD external hard drive<\/li>\n<\/ul>\n<\/li>\n
- Store Source images when running with Vision Job<\/li>\n<\/ol>\n
- \n
- \n
- \n
- TM SSD external hard drive<\/li>\n
- Network Service<\/li>\n
- Save Image Component<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
\nWhen\u00a0 choosing the\u00a0 second\u00a0 way\u00a0 to\u00a0 store\u00a0 Source\u00a0 images,\u00a0 you can\u00a0 Crop\u00a0 ROI\u00a0 in\u00a0 the\u00a0 Dataset\u00a0 of\u00a0 version 2.12 Training\u00a0 Server
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/06\/store-image.png\")
<\/a><\/p>\n
\nOpen source image
\n![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/07\/opensourceimage.png\")
<\/a><\/p>\n
\nCrop ROI.\u00a0 The\u00a0 ROI will be labeled as unlabeled after the crop.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/crop_3.png\")
<\/a><\/p>\n
\nTrain models with TM AI+ and import AI models into robot testing<\/strong><\/h2>\n
TM AI+ operation<\/h3>\n
Save the images of the inspection processes above into two datasets, one for OK and another for NG, as shown below.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/data.png\")
<\/a><\/p>\n
\nTM AI+ is built on the company’s internal server. Any device that can connect to the company’s intranet can use TM AI+ through a browser, and there is no restriction on the system for operation. This article uses TM AI+ Training Server 2.12.2200 as an example.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/ailogin.png\")
<\/a><\/p>\n
\nThis is the manegerment screen of the Dataset, where we can import our datasets.
\n![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/07\/datasetmanagerment.png\")
<\/a><\/p>\n
\nCreate a classification dataset.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/07\/name-and-save.png\")
<\/a><\/p>\n
\nClick Edit to create a dataset.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/07\/click-edit.png\")
<\/a><\/p>\n
\nCreate labels and upload the corresponding datasets.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/07\/create-label.png\")
<\/a><\/p>\n
\nThis is the project management screen, where you can create our projects.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/07\/add-project.png\")
<\/a><\/p>\n
\nAdd a project, name it and select Classification.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/07\/name-project.png\")
<\/a><\/p>\n
\nChoose the appropriate model and dataset according to the task scenario.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/07\/select-model.png\")
<\/a><\/p>\n
\nAfter adjusting the parameters, start training the model.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/07\/train-model.png\")
<\/a><\/p>\n
\nAfter the training is completed, you can use summary to check the performance of the trained model, and use Heatmap to analyze whether the information learned by AI is the feature we want to compare. Then this article will analyze each detection result and perform analysis.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/07\/summary-model.png\")
<\/a><\/p>\n
\nTM AI+ – Is equipped with filter ring<\/strong><\/h3>\n
Observing the training progress, the accuracy of each training session is almost 100 percent, indicating that the model for detecting the filter ring is more stable.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/ai_ferrite.png\")
<\/a><\/p>\n
\nBy using the heatmap, we can see that AI has detected the filter ring on the line, indicating that features learned by AI are reasonable. We import this model into the robot for testing, and it can stably distinguish whether there are filter ring.<\/p>\n
Then we switch to another robot and environment for testing and it can also stably distinguish, without changing the model.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/ring-heatmap.png\")
<\/a><\/p>\n
\nTM AI+ – Is the plugin actually inserted<\/strong><\/h3>\n
Observing the training progress, the accuracy of multiple training sessions is 100 percent, indicating that the model for detecting whether the plug is properly inserted is more stable.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/ai_insert.png\")
<\/a><\/p>\n
\nBy using the heatmap, we found that the features learned by AI focused on the small black hole at the top. If it is not properly inserted, the black hole at the top will be particularly obvious and will be judged as NG. The result is as we expected. The heatmap of the NG situation also pays attention to whether the black hole at the top is too obvious, or whether the plug is completely detached. After importing the model into the robot, it can also run smoothly in various situation.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/insert-compare-1.png\")
<\/a><\/p>\n
\nThen we switched to another robot and environment for testing, and found that the situation of complete detachment was easily judged as OK. We guessed that the change of light source caused the wiring to be unclear, and mistakenly judged the situatin of the wire head lying flat on the top of socket. So in this environment, we added about 20 more NG situations, and also added about 20 more OK situations to adapt to the environmental changes.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/misjudge-insert.png\")
<\/a><\/p>\n
\nTM AI+ – whether there is a screw in the screw hole_top shot of the IPC<\/strong><\/h3>\n
Observing the training progress, the accuracy of the training session is mostly 100 percent, indicating that the model for detecting whether there are screws in the screw holes under the condition of shooting the IPC from above is stable.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/ai_upscrew.png\")
<\/a><\/p>\n
\nBy using the heatmap, we can see that AI can learn the feature of whether there are screws on the holes under any light source. After importing the model into the robot, the test is also very stable.<\/p>\n
Then we switched to another robot and environment for testing, and it can also stably distinguish, without changing the model.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/ai_upscrew_new-1.png\")
<\/a><\/p>\n
\nTM AI+ – whether there is a screw in the screw hole_side shot of the IPC<\/strong><\/h3>\n
Observing the training progress, the accuracy of the training sessions is also mostly 100 percent, indicating that the model for detecting whether there are screws in the screw holes under the condition of shooting the IPC from the side is also stable.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/ai_side-1.png\")
<\/a><\/p>\n
\n- \n
- Observing the heatmap, the features that AI focuses on are indeed on whether there are screws in the screw holes. But when importing the model into the robot for testing, some angles of the screw holes are easily affected by the ROI selection and recognized incorrectly, indicating that this model is not stable enough and the data is not enough.<\/li>\n
- It is recommended to consider the image of screws taken from above and train them together to obtain a relatively stable model. You can also combine the previous data on screw locking to train a more robust model. In this way, this model can be applied to most of the screw holes on this IPC.<\/li>\n
- Considering the model trained with more data, it can also stably judge whether there are screws in the screw holes after switching the robot and environment.<\/li>\n<\/ul>\n
\nTM AI+ – whether the cable is connected correctly<\/strong><\/h3>\n
Using the above-mentioned image acquisition method for the first training, the training progress is less stable than the previous situations.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/ai_cable.png\")
<\/a><\/p>\n
\nObserving the heatmap, we cannot determine whether the situations of OK overlapping and CROSS have been learned by AI. We need to import the robot for actual testing.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/ai_cable_1.png\")
<\/a><\/p>\n
\n1.After importing the robot testing, we can find that the two crossed lines under CROSS are easy to judge as OK, while the three and four crossed lines have no problem.<\/p>\n
2.And under the NG situation, the lines that are not inserted side by side are also easy to be judge as NG.<\/p>\n
3.Under the OK situation, if there is overlapping, it is also easy to judge as CROSS.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/wiring-compare.png\")
<\/a><\/p>\n
\nAfter collecting about 20 images of the misjudged situations and retraining, the problem still could not be improved, and even failed to converge to a more correct result.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/ai_cable_3.png\")
<\/a><\/p>\n
\nSo we changed the model selection from Regular to Ultra for training, performed data augmentation, and some minor parameter adjustments, to make it consider more variations.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/ai_cable_4.png\")
<\/a><\/p>\n
\n1.Epochs increased from 30 to 100, giving it a chance to find a convergent result.<\/p>\n
2.Batch Size is set smaller, so that it will not see too many possibilities at the same time, and cause it to fail to converge.<\/p>\n
3.Learning Rate is slightly adjusted lower, so that it has more chance to reach a lower Loss.<\/p>\n
![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/ai_cable_5.png\")
<\/a><\/p>\n
\nAfter the adjustment, the training progress has found a lower Loss, and some 100 percent Accuracy has appeared, but it is still not certain that the trained model is reasonable.
\n![\"\"](\"https:\/\/tm-robot.oss-cn-hongkong.aliyuncs.com\/wp-content\/uploads\/2023\/02\/ai_cable_6.png\")
<\/a><\/p>\n
\nThen import this model into the robot test, which improved the following issues<\/p>\n
- \n
- Store ROI image in AI Classification\n
- Is equipped with filter ring\n