Comments on Black Boxes
Your process of opening the "Black Box" of Neural Networks is truly insightful. By creating a straightforward Neural Network from scratch and examining its essential mechanics, you've revealed a domain filled with potential. This journey transcends a mere academic exercise; it's a rich, multifaceted experience characterized by an iterative two-phase approach. The real challenge was in synthesizing knowledge from various fields, including creativity, efficiency, mathematics, and developing a real-world model from scratch. The reflective nature of this work will help other readers understand the complex structure that supports machine learning. Great job Darren. Bravo!
2024-08-21 23:44:35
Great comment Jeff! I know you are much smarter than an AI but it's nice to know Perplexity.ai agrees with you. Here's how it responded: "Overall, the Author’s approach stands out for its comprehensive, ground-up construction combined with innovative visualization techniques and a focus on real-world applicability. This method potentially offers a deeper understanding of Neural Network mechanics and behavior compared to approaches that rely more heavily on pre-existing tools or focus solely on theoretical aspects."
2024-08-22 17:01:03
Enjoyable read and cool way to tackle the problem of understanding neural networks. I especially like the way that you've compared different optimization approaches, giving insight into the pros and cons of gradient descent. I'd love to hear more details about how you applied the Repetitive Nearest Neighbor approach to weight optimization, and how well you think it would work on larger networks. I would expect Gradient Descent would win out on larger networks, but if that's not the case it would be a very interesting result. I've also been looking for good visualization techniques. I've been focusing on the internal dynamics of the neural network model. You might be interested: Better Ways to Monitor NNs
2025-05-08 05:47:00
Malcolm, Thanks for the insightful comment and link to your article. I was pleased to discover you have a Medium channel with several articles of interest: Malcolm Lett on Medium I think you might be correct about "Repetitive Nearest Neighbor" working well on small ANNs but maybe not a technique that would scale. Plain "Nearest Neighbor" is just a Greedy technique, and depends on your starting point for optimization. "Repetitive" applies the Greedy technique to each of the starting points and takes the best result. So while it does not grow exponentially like "Brute Force" it does grow linearly, which could still be too much overhead. Other Traveling Salesman optimization techniques have impose much less overhead. I wonder if one of those would also outperform Gradient Descent, but still scale nicely.
2025-05-08 17:31:09By the way, Kolmogorov-Arnold Networks (KANs) are one attempt to allow individual units to learn more interesting response curves. It might fill the gap between generic solutions and the domain-knowledge approach you used.
2025-05-08 17:58:27