For some time I was fascinated by the correlation between Karl Popper writings in philosophy of science and some fundamental ideas in machine learning.
I first 'met' Popper when I was 20 years old and Daniel (a great mentor and friend that happens to be my uncle :) advised me to read "The Open Society and its Enemies". The Romanian society at that moment was still strongly marked by the communism years. The stories, the sufferings, the experience of totalitarianism were still alive around me and I could feel and understand them but in a confuse and sentimental way. Reading this book was a great experience because it opened my eyes to a rational account of what happened. It was like a scientific theory of tyranny, its causes and effects. It made me think and rationalize what I was feeling before.
Later on, skimming through "The Logic of Scientific Discovery" I realized that "The Open Society and its Enemies" was only an instance of a more generic view towards rationality in any scientific discipline. Popper introduced in 1934 (I marked the year to emphasize an idea later on) the theory of falsifiability, which shortly says that for a theory to be named scientific, it should offer the possibility to be negated in some clear conditions.
In a more detailed manner, Popper claims that for a discipline to be scientific (~ predictive of real world phenomenons, valuable to human knowledge) it needs to offer predictions which are potentially verifiable (and falsifiable) in an experimental manner. A good scientific theory will offer good predictions (like Einstein's general relativity - Popper's favorite example), a bad one, bad predictions. A non scientific theory will not offer a generic frame that will allow others to negate it (see astrology). If you want to know more here is an excerpt written by the author: Science as falsification.
Even later on, when first reading about machine learning I was struck by the similarity of Popper's ideas and the necessity in ML to have a hypothesis (that is obtained from after training on a training set) that is tried on a different set to test its generalization accuracy. Unless you have such a test set, you can't say that you learned a good hypothesis. It's the same thing as with scientific and non-scientific theories.
And my improbable personal knowledge circle was complete when skimming through Vladimir Vapnik's "The nature of statistical learning theory" I read: "Before continuing with the description of statistical learning theory, let me remark how amazing Popper's idea was. In the 1930s Popper suggested a general concept determining the generalization ability (in a very wide philosophical sense) that in the 1990's turned out to be one of the most crucial concepts for the analysis of consistency of the ERM inductive principle" (page 55). It seems remarkable (at least to me) how someone with no mathematical tools was able to come up with such an idea (Popper formalized it really well as Vapnik's acknowledges ; the two actually met at a certain point towards the end of Popper's life). And it is also an indication of how powerful philosophy can be.
Update: Thanks to David Corfield for pointing me to a much more informed opinion regarding the connection between statistical learning and Popper's ideas: "Popper, Falsification and the VC-dimension" by David Corfield, Bernhard Scholkopf, Vladimir Vapnik.
Friday, November 14, 2008
Thursday, November 13, 2008
Random Machine Learning thoughts
A part of my brain is still fighting the idea that if you don't make any kind of presupposition (if you don't have a bias whatsoever) then you won't be able to learn anything.
It's still strange that when you perfectly learn something then you are overfit and you won't be able to generalize in a good manner.
It's amazing for me that a 2 layer neural network can act as a universal continuous function approximator.
I feel there is a gap between the math in ML and the philosophy that ML should generate and that's kind of sad.
I keep wondering how come not everybody wants to study AI, machine learning and the like - I'm kidding, but deep inside I'm really stupefied :). A good friend asked me what would I do (professionally) if I would win a huge sum on the lottery (no danger here, I never buy tickets). And I felt really well thinking and saying that I would continue doing exactly what I am doing now.
I use the opportunity of an interesting post about the (lack of?) power of randomness to refer you to one of my favorite blogs: Overcoming Bias. (a blog at the intersection between philosophy, cognitive science, and a bit of AI).
It's still strange that when you perfectly learn something then you are overfit and you won't be able to generalize in a good manner.
It's amazing for me that a 2 layer neural network can act as a universal continuous function approximator.
I feel there is a gap between the math in ML and the philosophy that ML should generate and that's kind of sad.
I keep wondering how come not everybody wants to study AI, machine learning and the like - I'm kidding, but deep inside I'm really stupefied :). A good friend asked me what would I do (professionally) if I would win a huge sum on the lottery (no danger here, I never buy tickets). And I felt really well thinking and saying that I would continue doing exactly what I am doing now.
I use the opportunity of an interesting post about the (lack of?) power of randomness to refer you to one of my favorite blogs: Overcoming Bias. (a blog at the intersection between philosophy, cognitive science, and a bit of AI).
Saturday, October 11, 2008
NY Machine Learning Symposium
I was at the 3rd Annual Machine Learning Symposium today. This a (rather new) workshop that aims to bring together researchers from NYC area that work in machine learning.
It was a very nice experience. Besides the cool ideas and the presentations, I really appreciated the atmosphere, the openness of the people and the great view (see below).
From the about 30 student posters they selected 10 finalists (that had short but informative presentations) and the winner was a NYU team with the paper "Sample Selection Bias Correction Theory" , which was really interesting. If I understood correctly, they said that usually when a test set is generated for a learning problem some samples are not presented to the learner (because of a higher cost of the labeling for instance); thus a bias is introduced in the test sample. This bias is of course unwanted and should be corrected. They presented a theoretical analysis of sample selection bias correction and presented some techniques of how to do it.
Regarding the presentations, the most interesting (and sometimes a little scary) for me was that of Tony Jebara from Columbia. The idea was that
And the picture from outside the conference room (naturally the chairs were facing the wall, otherwise I surely wouldn't have been able to concentrate on the presentations):
It was a very nice experience. Besides the cool ideas and the presentations, I really appreciated the atmosphere, the openness of the people and the great view (see below).
From the about 30 student posters they selected 10 finalists (that had short but informative presentations) and the winner was a NYU team with the paper "Sample Selection Bias Correction Theory" , which was really interesting. If I understood correctly, they said that usually when a test set is generated for a learning problem some samples are not presented to the learner (because of a higher cost of the labeling for instance); thus a bias is introduced in the test sample. This bias is of course unwanted and should be corrected. They presented a theoretical analysis of sample selection bias correction and presented some techniques of how to do it.
Regarding the presentations, the most interesting (and sometimes a little scary) for me was that of Tony Jebara from Columbia. The idea was that
"With machine learning algorithms applied to these human activity graphs, it becomes possible to make predictions for advertising, marketing and collaborative recommendation."(citation from speaker abstract). They used advanced algorithms to: 1. form a graph from sparse (GPS in this case) data (based on an approximation of k-matching algorithms - of which I just found out today and seems really cool), 2. partition the graph in a 'meaningful' way, 3. interpret the information in terms of what behaviors groups of people have so as to better match (for instance) them with advertisers. This was the scary part in the sense that they tracked people movements in a city what restaurants they went, when, etc (of course the information was made anonymous but still it felt strange to see an animation with clusters of people doing different stuff) - I wasn't familiar with most of the theoretical part but was interesting to learn more.
And the picture from outside the conference room (naturally the chairs were facing the wall, otherwise I surely wouldn't have been able to concentrate on the presentations):
Saturday, September 20, 2008
A new beginning
It's been a long time since I gave up at writing something here. It was for good reasons.
Even though I didn't write a lot (about 10 articles, and all of them very far from the quality or the content I wished when starting this), this blog was one of the factors that made me realize an important thing: that I wasn't following my dreams. Trying to write interesting AI related things here, I discovered that I didn't have the time to really study, understand and communicate them, except on a very superficial level. And it was frustrating. So this, among other things, led me to decide to want to return to the academic realm. And after GREs, applications and the usual track of a foreign applicant, this autumn I started a PhD in Computer Science at Rutgers University with the hope to study in depth machine learning and reinforcement learning. Hopefully, I'll have more interesting things to write from now on.
Even though I didn't write a lot (about 10 articles, and all of them very far from the quality or the content I wished when starting this), this blog was one of the factors that made me realize an important thing: that I wasn't following my dreams. Trying to write interesting AI related things here, I discovered that I didn't have the time to really study, understand and communicate them, except on a very superficial level. And it was frustrating. So this, among other things, led me to decide to want to return to the academic realm. And after GREs, applications and the usual track of a foreign applicant, this autumn I started a PhD in Computer Science at Rutgers University with the hope to study in depth machine learning and reinforcement learning. Hopefully, I'll have more interesting things to write from now on.
Tuesday, June 19, 2007
Most spectacular AI applications
Below I tried to compile a short list with the most impressive AI applications (any other visions regarding this are welcomed). I find it fascinating that AI is far from being dead or dead-locked as some say and instead has a direct impact on our lives. It is most of the time hidden behind things that become normal with usage, but that does not mean that it's not there.
Mars Spirit and Opportunity Rovers. Why are they impressive? Because they wonder semi-autonomously on a different planet for 3.5 years.
What is interesting from an AI point of view is the control system - which is much better than the one for the Sojourner's mission. The rovers have stereo cameras and generate 3D maps that allow them to avoid any obstacles by choosing the "path of least resistance". The human drivers usually give them a target point situated at "big" distances (hundreds of meters) and the rovers decide how they should get to that point.
Google
It is impressive because it is the first large scale successful implementation of complex machine learning techniques. I've always considered Google not so much the success of a Web 2.0 approach but more of an AI 2.0 approach.
AI is the "hidden" secret of Google search. They attract the best machine learning scientists (the best example is that Peter Norvig is Google's Director of research).
DeepBlue
It is impressive because it was the first computer to beat to world chess champion.
Although its main characteristic was brute force (capable of analyzing 200.000.000 moves a second) he also incorporated heuristics that allowed him to beat Kasparov.
It marked the beginning of an era - the moment when people realized that computers could outsmart them at things that were considered linked to highest levels of intelligence.
Resources:
It is impressive because it is the first large scale successful implementation of complex machine learning techniques. I've always considered Google not so much the success of a Web 2.0 approach but more of an AI 2.0 approach.
AI is the "hidden" secret of Google search. They attract the best machine learning scientists (the best example is that Peter Norvig is Google's Director of research).
DeepBlueIt is impressive because it was the first computer to beat to world chess champion.
Although its main characteristic was brute force (capable of analyzing 200.000.000 moves a second) he also incorporated heuristics that allowed him to beat Kasparov.
It marked the beginning of an era - the moment when people realized that computers could outsmart them at things that were considered linked to highest levels of intelligence.
Resources:
- http://marsrovers.nasa.gov/home/index.html
- http://marsrovers.nasa.gov/technology/bb_software_engineering.html
- http://en.wikipedia.org/wiki/Spirit_rover
- http://www.research.ibm.com/deepblue/learn/html/e.8.1.shtml
- ftp://ftp.cs.yale.edu/pub/mcdermott/papers/deepblue.txt
- http://en.wikipedia.org/wiki/Deepblue
Tuesday, March 27, 2007
CS on the map of science
Through Slashdot I got to a Map Of Science - an interesting view of the connections between 800k published papers in 776 scientific paradigms (see the complete image here).
What seemed really interesting to me were the closest domains to Computer Science: Math, BrainResearch, Astrophysics and Social Sciences. Although the first three don't come as a surprise, the last one is a bit unexpected.
What seemed really interesting to me were the closest domains to Computer Science: Math, BrainResearch, Astrophysics and Social Sciences. Although the first three don't come as a surprise, the last one is a bit unexpected.
Sunday, February 4, 2007
AI academic research
The idea for this post came to me after reading a great series of 3 articles on Machine Learning Thoughts about how it is to be a scientist (read here, here and here).
Do you want to do academic research? In AI maybe? At what level? (i asked myself these questions at a certain point and all the ideas the follow are only my personal opinions).
Q1: What country?
A1: USA seems the good place to be at the moment for AI research. Top level universities, top level conferences, top level firms where you can project ideas into reality and Silicon Valley where you can make your startup and change the world.
Q2: What universities / research groups?
A2: You can find a lot of rankings on the internet - general, engineering, computer science and Artificial Intelligence. For AI the ranks are old, but they give a good idea of places where interesting things happen. The only place that is missing from there is Georgia Tech. Of course in the end it matters if you find a place with a great teacher (they are not all at MIT or Carnegie) that does what you like, but your chance of finding this are much higher at the first 10 schools. Be it robotics, machine learning, text mining you can find in those universities the majority of people that make things moving in AI research.
I was impressed since i was in high school of the incredible number of robots from MIT and then in university when i heard of seemingly science-fiction projects from the same university.
Q3: How do you get in such great places?
A3: I am not an American so i only know about experiences of foreigners. You have to a good GPA (unless you are really great in other aspects), good GRE / Toefl scores, nice recommendations and a good intuition of what those that make the selection want to hear in a statement of purpose letter. Oh yes, and if possible research activity (publications at conferences, in journals).
Q4: How do you publish?
A4: Well, that's tough. You have to have an idea of what you like and try to touch the borders in the specific domain. When you do try to push it a little further and try to project that into a paper (just my opinion remember :). Then (or before) find a professor with similar interests and try to get some advices and maybe a collaboration.
Q5: Where do you publish?
A5: It depends a lot on how good your idea / paper is. You can find here the ranks for AI conferences (remember, publishing at AAAI, ICML or NIPS might be as hard as being accepted at MIT, Carnegie Mellon or Berkley).
Your chance of being accepted? See a statistic of the last years acceptance rate. It's also a good idea to look at least at the titles of the papers accepted at the conference / journal you want to apply to in the previous years to get an idea of the "hot" and preferred topics.
Q6: Is it possible to do research directly in a company as to not "lose" so much time with a PhD?
A6: I don't know since i don't have any experience with that (if you know, tell me). Logically, it would seem improbable that a top notch industrial research department would choose a John Doe instead of a bright PhD from one of the first 10 universities, so i would be inclined to say no, it's not very probable for this to happen (but again, i might be very wrong).
I'm sure there are a lot of other questions, but in the end the most important ones are: do you want to do research? is this something that suites you? are you ok with spending some years without earning so much money as in the industry and to take the risk of not making anything significant instead? are you passioned of the ideas you have? If you answer yes to all of these then it might be a good idea to start looking for an university :).
Do you want to do academic research? In AI maybe? At what level? (i asked myself these questions at a certain point and all the ideas the follow are only my personal opinions).
Q1: What country?
A1: USA seems the good place to be at the moment for AI research. Top level universities, top level conferences, top level firms where you can project ideas into reality and Silicon Valley where you can make your startup and change the world.
Q2: What universities / research groups?
A2: You can find a lot of rankings on the internet - general, engineering, computer science and Artificial Intelligence. For AI the ranks are old, but they give a good idea of places where interesting things happen. The only place that is missing from there is Georgia Tech. Of course in the end it matters if you find a place with a great teacher (they are not all at MIT or Carnegie) that does what you like, but your chance of finding this are much higher at the first 10 schools. Be it robotics, machine learning, text mining you can find in those universities the majority of people that make things moving in AI research.
I was impressed since i was in high school of the incredible number of robots from MIT and then in university when i heard of seemingly science-fiction projects from the same university.
Q3: How do you get in such great places?
A3: I am not an American so i only know about experiences of foreigners. You have to a good GPA (unless you are really great in other aspects), good GRE / Toefl scores, nice recommendations and a good intuition of what those that make the selection want to hear in a statement of purpose letter. Oh yes, and if possible research activity (publications at conferences, in journals).
Q4: How do you publish?
A4: Well, that's tough. You have to have an idea of what you like and try to touch the borders in the specific domain. When you do try to push it a little further and try to project that into a paper (just my opinion remember :). Then (or before) find a professor with similar interests and try to get some advices and maybe a collaboration.
Q5: Where do you publish?
A5: It depends a lot on how good your idea / paper is. You can find here the ranks for AI conferences (remember, publishing at AAAI, ICML or NIPS might be as hard as being accepted at MIT, Carnegie Mellon or Berkley).
Your chance of being accepted? See a statistic of the last years acceptance rate. It's also a good idea to look at least at the titles of the papers accepted at the conference / journal you want to apply to in the previous years to get an idea of the "hot" and preferred topics.
Q6: Is it possible to do research directly in a company as to not "lose" so much time with a PhD?
A6: I don't know since i don't have any experience with that (if you know, tell me). Logically, it would seem improbable that a top notch industrial research department would choose a John Doe instead of a bright PhD from one of the first 10 universities, so i would be inclined to say no, it's not very probable for this to happen (but again, i might be very wrong).
I'm sure there are a lot of other questions, but in the end the most important ones are: do you want to do research? is this something that suites you? are you ok with spending some years without earning so much money as in the industry and to take the risk of not making anything significant instead? are you passioned of the ideas you have? If you answer yes to all of these then it might be a good idea to start looking for an university :).
Subscribe to:
Posts (Atom)
