AI

AI, Reinforcement Learning

Reinforcement learning

Reading Time: 2 minutes

Ever since the Deepmine Alpha Go paper came out, I have always been fascinated by what reinforcement learning could do. In my opinion, it’s the closest branch of AI to Artificial General Intelligence. Because no matter what your goal is, as long as you determined the reward correctly, it will eventually help you to find the “optimal” solution. What it lacks, perhaps, is that it takes a long time to learn. For example, human babies don’t have to stumble millions of times to learn how to walk. And it’s even more evident for a deer, where they pretty much know how to walk after they were born. Now, that is not too say evolution didn’t give some prebuilt model for the deer to use, but maybe we can make some pretrained model for AI to know how to walk, like the modules they insert into Neo to give him fighting abilities.

When I set out to learn reinforcement learning, there wasn’t much formalized material. There is Sutton and Barto’s book, but I found it was and still written for experts. The exercises they give are necessary for me to understand what’s going on, but there is no explanation given. There is too much “for obvious reasons” for me to understand some basic concepts, like how do I calculate state value for each grid. There is a formula for it, but no detailed walkthrough to how to assign those values. Then there are David Silver’s lectures, which shined more light for me than the book, but there is also too much detail left unexplained. So, I have searched the internet for some tutorials and finally landed on this medium article. https://medium.com/free-code-camp/an-introduction-to-reinforcement-learning-4339519de419. It has enough little details for me to work out how to get to individual values, starting from what’s a Q table and how implement it in python. I highly recommend going through the tutorial and exercises on Google Colab and make sure you understand how the environment is used in the Gym package.

Anyway, I’m still working through all of the code and understanding it. I also recommend a Lego Mindstorm set to give you a physical manifestation to work with. It can help you familiarize with how to apply it in the physical world. The basic robotics with a stable, not bipedal, robot, also makes learning easier. Make sure you use a customize python package like ev3dev so you can control the robot with python scripts. https://www.ev3dev.org/ Now that’s how you have endless, time sucking fun!

AI

AI learning material

Reading Time: 2 minutes

Recently, I have received many requests about how I learned Neural Networks. Since this is so new that not that many Universities offer a class in it, I thought I should put together a list of learning material I found useful. Hopefully other people will find it useful too.

  1. I’m an audio learner and love to listen to people. So Andrew Ng’s Coursera course was good for me. https://www.coursera.org/specializations/deep-learning. Take all four if you have time and try to understand the exercises and not just manipulate the program to get the correct answer. Actually understand the Math behind it.
  2. Ian Goodfellow’s books cover the math background you need and he shows you how to apply it. https://www.deeplearningbook.org/. This is a good reference book if you find yourself lost in some concept a paper mentioned. The dreaded, “for obviously reasons, …”
  3. If you find you want some statistics background, I read this book, http://www-bcf.usc.edu/~gareth/ISL/. There is another more bible like statistics book, but if I cannot understand it, I’m not going to suggest it to anybody.
  4. Cornell University have an open archive for all sciences. And you can find the specific journal articles related to AI, Computer Vision, Speech, and NLP. https://arxiv.org/. It is intimidating in the beginning to read papers, just from the amount of jargons people use. But after you read several it’s not that bad. Otherwise, refer to Ian’s book or just google it.
  5. MIT online course videos are good for general background in Math and CS. Again, I prefer listening to lectures much more than reading books. I just fall asleep too much.
  6. Stanford class notes is where I started: http://cs231n.stanford.edu/. Image recognition is easier to understand but hard to master.
  7. Grant’s video on youtube has the best visualizations. And they are the best linear algebra video I have ever seen on transformation. You are going to need linear algebra. It speed up your computation by a ton, if nothing else. https://www.youtube.com/watch?v=aircAruvnKk&list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi
  8. I would say most import of all, find a paper with code and just play with it. You can put code on Google Colab and get free GPU. https://paperswithcode.com/

Secret: Use Twitter to follow the experts in AI. Build it up to specialize in the field you like. e.g. reinforcement learning. I will get you started with some famous names: @geoffreyhinton, @ylecun, @AndrewYNg, and @goodfellow_ian.

AI, Misc

How I went from a Neuroscientist to an AI practitioner – Part II

Reading Time: 4 minutesAfter studying further about what neural networks can do, I realized this is what people are going to do in the future. For everything. And I know I got to get a piece of it. So I started to learn the basics. At the time there was not much training or tutorials. AlexNet was widely used and I ran across a tutorial on using Matlab to make VGG16 convolutional NN. I had pretty good video card, but it wasn’t able to run the Matlab package. I had my eyes on the upcoming Nvidia 1070 cards. I had to wait for it to release and I new it was going to be hard to get.

In the mean time, I figured out it doesn’t have to run GPU. And I can start learning the basics before I can get my hands on the new cards. I searched around. Most of the tutorial have fragmented information, and I feel I have no grasp on the material and did not really understand anything. Then I found Stanford’s CS213n class notes, and felt I hit the gold mine. It was technical and theoretically involved, but it explained how CNN model pixels and many of the CNN tools like weight initialization and dropout.

Once I got a sense of how CNN works, I joined Kaggle’s competitions to actually ran my own models. I tried the simple ones like Titanic morality prediction that require heavy feature engineering, but quickly realized that’s not the way to do things, because it required too much manual work and introduced bias. Then I saw the competition where State Farm want to categorize pictures taken inside of cards for what drivers were doing. That required me to use opencv for image pre-processing and categorized pictures using training and test sets. I didn’t end up having a super high ranking, but I learned various ways of doing it from people’s published kernels.

Fast forward several month, and I got my hand on a Nvidia 1070. Popped that puppy in the tower and sped up the models by a huge chunk. Of course, by this point, I knew Python is the way to go and used Theano and Tensorflow for making the models. I played with some public data and started a project to estimate house price in NYC, because everybody was doing it. I tried out for an interview with Insight Data Science. I prepared my projects and seemed to know what I talked about. I was told that I was on the board line and barely missed the cut, but maybe she was just being nice.

No problem, I will try again. On the third real try for Data Incubator, I finally answered enough questions in Python and got an interview. After a group interview with 6 other applicant, I got an offer for the paid, second tier fellowship program. Supposedly, I wasn’t as good as what they are looking, but if I pay for the fellowship, I can tag along. It was a large amount of money, even for NYC standards, but I had other considerations.

My Professor was about ran out of funding, at least for me. So I was about to be out of a job, unless I want to find another postdoc position. I started looking for industry jobs, but wasn’t successful. Just one week earlier, I got an offer to join a doctor’s office to do In Vitro Fertilization stuff. I applied because I have many family connections in that field, and I actually know how to do the stuff. It was a safe job with limited upside. My wife encouraged me to turn it down and pursuit my dream in AI. And I thank her so much now. I would be doing the same thing day after day to earn a living if I took that job. But instead, I’m learning new things everyday and loving it.

Coming back to my decisions, it was tough to turn down offers when I about to be unemployed. Again, my wife agreed to “take care of me” for 6 month, since my research said that’s the average time of unemployment for changing careers. It gave me a piece of mind, and let me focus on what to learn. With all my personal affaires lined up, I accepted the layoff and prepared to be workless, payed a hefty amount for the fellowship, and doubled downed on my future.

The fellowship was enjoyable. Hard, but fun. We were squeezed in a box for at least 8 hours a day. Studied as much as we could, did a lot of programming, prepared for interviews, and met some potential employers. It was a good variety of topics, and I was still talking about it 6 months later. As fun as it was, time flies. But the end of the session, 2 people got offers. Some are hopeful, I wasn’t one of them.

When the time of workless came, strangely, I wasn’t afraid. Moved out of NYC, we ranted an apartment in a suburban area. I kept the discipline of applying jobs everyday and interviews kept happening. A few showed interest, but ultimately, none of them went through.

Finally, a family member helped me to find a job, but it a rollercoaster. I got a phone interview a year earlier. A statistician asked about what models I made before and some other technical questions for half an hour. No feedback or anything after that. Seven months later, I got an email about scheduling onsite interview out of the blue. Of course I went. It was a grueling 8 hour day interview. Mostly behavior questions, with a few stat questions throwing in between. Then nothing. Mostly mentioning there is no position opening right now.

A month later, I got an email from a different department asking to schedule a phone interview. This time more questions on neural networks and programming. I knew I wanted whatever job they have. But again, no feedback on job or no job. A month later, same group of people called again and simply asked do I want the job. They knew I wasn’t sold on programming, but they are willing to teach. I had to go for an onsite, but from what I felt, it was just a routine I had to go through. The decision was already made.

Everything was what I though it would be and better. The rest is history. Now I’m working on applying the latest AI algorithms to industry related tasks. Some people asked, do you feel overwhelmed by doing more than 20 projects in one year. I responded to that: I was never too busy to do the things I love.

So, for people who are looking to change their career to AI, or any career you love, don’t give up. The road may be long and scary, but we only live once, and you should go for broke.

AI, Misc

How I went from a Neuroscientist to an AI practitioner – Part I

Reading Time: 5 minutesCareer change is often thought of as a tough act to pull off. It is especially difficult to transition from a highly specialize field to another. However, everything happened for a reason, whether you know it or not. I started college as a computer science major for about a year, then I was forced to change to Biology, due the Dot Com bubble around the year of 2001. My parent heard the news that internet companies fell one-by-one. And they thought my computer science is not a traditional, stable career path to follow. (Of course they were proven wrong many years later).

When I go home for the summer, my parents and I had a talk. And they convinced me that I should change my major to Biology and possibly becoming a doctor. At that time, I was weak-minded, and didn’t want to displease my parents. And the fact that my college room and board was paid by my parents didn’t help. I reluctantly changed my major and started taking biology related class. I didn’t waist too much time since I took some AP class in high school and was able to test out of Chem 101 and Physic 101. I have to say that Biology was not something I hate. In fact, it was my best interest before my dad bought a computer when I was 11. So, I chucked along with the courses. Got my fair share of As and Bs, and of course, a bad grad for Organic Chemistry.

Bioinformatics was a hot topic at the time. With my limited programming skills, I was able to get a fellowship in computational biology from Howard Hughes Medical Institute. With a few classes in protein folding and predicting tertiary structure from protein sequence, I learned the basic knowledge to BS my way through college. Then I met my undergraduate advisor, who is interested in programming and nifty projects. We painted the rat’s paws with colored markers and videotaped their footsteps through Plexiglas from underneath. We wrote a C++ plugin in iMovie that tracked the footsteps and how faster they turned following various experiments. It was a blast working with him and it seemed like a fun project for him. I wrote my undergraduate thesis on the topic, although it seems like unpublishable junk looking at it now.

Then it’s time to prepare for the dreaded MCAT. I knew I wasn’t going to study enough on my own, so I paid a hefty amount to join a prep class. The teacher is a current Med school student that got a good score. He showed us how to prepare and do practice tests. But he also said Med school is not for everyone because he constantly practices involuntary fasting and became an insomniac. I did as many practice test as I could, but because I was not enthusiast about it, I didn’t get a good score. The eight-hour test was not that bad, as I had those tests then I was in China, as middle schooler. It was good enough for me to get into a D.O. school, but didn’t want to be a primary doctor. The chance of specialize in Radiology or Neurology was below 10%. And it was not good enough for me get a M.D. I later found out that being Asian did not help either. I wasn’t going to be one of those Hispanic female applicants who got in with a 21 on MCAT.

I wasn’t disappointed when I found out I don’t have much of a chance with a dismal 27 on MCAT.  I was lost for a while. I got a simple gross room processing job at a pathology lab, then applied for graduate school. With my research background in college and honor with undergraduate thesis, it wasn’t too hard to get offers from multiple schools. It was years later I found out there is a good reason for those programs to offer stipend, because there is no job prospect even with a Ph.D. So, then I choose the best state school I can get into and started the long, long journey of graduate school.  Without too much detail here, I suffered through 7 years of graduate school, with many sleepless nights, and got my degree. I did learn a lot of things during that period, mostly built my resilient character, but also learned what hell was like. Perhaps it wasn’t like hell physically, but it was definitely mental hell.

With no marketable skill outside of rat brain and spinal cord surgery, my only choice was to become a postdoc and give a shot at becoming a faculty. There were two ways of finding such jobs. One is to send CV (resume) to every professor who is trying to find a postdoc. The other is focus on the lab you like to join, and write custom, specific letter to the few labs you are looking for. I was told the second option is how you find jobs in the industry, and I know people who got jobs that way. So I tried that. I researched the hand-full of labs I want to go to, read their papers, and thought about specific experience I would like to try if I joined their lab. I asked other people to proofread my email. And… Nothing. No answer. No rejection emails.

So I was force to go to the first option. I send out as many applications I could. Finding every ad on Science, Nature, or any other journal or website I can get my hands on. Three people returned my email. One said her research area is too different. One was really nice and asked for a follow-up skype interview. And one asked for a phone interview. The second guy talked about his research and promised he was going to take me. He described the surrounding area and planned out where I can rent to be closest to the lab. Then no contact for a month. During this time, the third guy called. We talked about the projects I will be working on and asked about what I can do. He liked my surgical skills, and offered me a position the next day. He didn’t give me much time to think, so I talked to some friends and took the job. It was a world class institution, and I didn’t have much of a choice. Two days later, the second asked for another skype session, but had to reject. “Dude, where were you for a month?”

Arrived at a major metropolitan for my job, I started to experience some new things. The professor like my surgical skills and promised to help me publish some papers soon. I got to do a little more programming than my graduate work and learned more concepts about signal processing. I had to learn how to apply linear algebra and partial differential equations to solve research problems. This was beneficial for getting an AI career. Big data became the hot topic, and so did Neural Networks. People started talking about data scientist jobs and I started looking in that direction as well. I looked for various boot camps in my area and tried out the “free” ones. They are hard to get into and I either failed initial interview or didn’t pass the screening process. But I kept trying for 3 years.

One day, DeepMind published their paper on Alpha Go. It was amazing! I used to play Go as a child. Although I never even got to 1 Dan at the amateur level, I could beat the best computer simulation easily. I knew this is hard to program. So, when the paper came out, I was blown away, but skeptical. Of course, I didn’t know what a reinforcement network is and only pretend to understand how it work. Then they announced they are going to play with Lee Sedol, a top ranked player in the world. Nobody though they would win. I stayed up 3 am in the morning to watch all 5 games. When the first game finished, my mouth dropped. I know Lee didn’t have a chance. The fact that he won the 4th game was nothing less than miraculous, and shows Alpha go still had weaknesses.

This is the day I realized AI is the future. This is the day I decided I want to do this for the rest of my life …

AI

Intelligent search

Reading Time: 2 minutesRecently, I started working on a business problem, where people want to search relevant material through many sources of documents by asking a question. This can be solved using model such as DrQA, with the data set similar to SQuAD. However, in order to use this model, the training data needs to be in question and answer pairs. This requires a lot of preprocessing to build up a good data set.

So the next thing we did is to use Solr to index the database first, creating links to the source and raw text of the source document. Searching by the question against the content of the Solr database, we first find the relevant document.

Once we find the relevant document, we first try to use machine comprehension function of AllenNLP to find the answer in the document. However, because AllenNLP use attention model, when the document is large ( > 90,000 characters), the model require too much memory.

So we tried Google’s Talk to Book approach. Using Universal Sentence Encoder by tensorflow, we separated the document into pages or paragraphs, and find the most similar page or paragraph to our question. This did not work because our question is in a question form and it is structurally different from the source document content. So we can either compare using a statement form to the document, or convert the document into many questions, and find which paragraph is the most similar to our question.

We tried the first option and used a statement to to find the most similar paragraph. It work fine and the 2nd most relevant answer was the one we are looking for. But from a user perspective, we cannot limit the user to not search in a question format. So we used Spacy and filtered out STOP WORDS, which include things like “what”, “how”, “do”, “I” and all the frequent words. This essentially turn question or statement into keywords we can use for search.

Another improvement we made was pulling top 3 most relevant document, parsed every paragraph from those documents and find the most relevant paragraph, regardless of where it came from. In this way, we able to rank the most relevant “paragraph” from multiple sources, and improve the search result as a whole.

View More