Gradient Descent for Linear Regression

Gradient Descent is defined as one of the most commonly used iterative optimization algorithm of machine learning to train the machine learning and deep learning models. It helps in finding the local minima of a function.

The best way to define the local minima or local maxima of a function using gradient descent is as follow:

  • If we move towards a negative gradient or away from the gradient of the function at the current point, it will give the local minima of that function.
  • Whenever we move towards a positive gradient or towards the gradient of the function at the current point, we will get the local maxima of tat function.

This entire process is known as gradient descent, which is also known as steepest descent. The main objective of using a gradient descent algorithm is to minimize the cost function using iteration.

Gradient descent Working

Before starting the working of gradient descent, we should know some basic concepts to find out the slope of a line from linear regression. The equation for the simple linear regression is given as:

Y = mx + c

Where ‘m’ represents the slope of the line, and ‘c’ represent the intercept on the y-axis.

  • At the starting point, we will derive the first derivative or slope and then use a tangent line to calculate the steepness of this slope. Further, this slope will inform the updates to the parameters (weights and bias).
  • The slope becomes steeper at the starting point or arbitrary point, but whenever new parameters are generated, then steepness gradually reduces, and at the lowest point, which is called a point of convergence.

Role of Learning Rate

The main objective of gradient descent is to minimize the cost function or the error between expected and actual. To minimize the cost function, learning rate plays a crucial role.

Learning Rate is defined as the step size taken to reach the minima or lowest point. This is typically a small value that is evaluated and updated based on the behavior of the cost function.

If the learning rate is high, it results is large steps but also leads to risks of overshooting the minima. At the same time, a low learning rate shows the small step sizes, which compromises overall efficiency but gives the advantage of more precision.

Popular Posts

Spread the knowledge

70 thoughts on “Gradient Descent for Linear Regression

  1. Thank you for your post. I liked reading it because it addressed my issue. It helped me a lot and I hope it will help others too.

  2. Thank you for posting this post. I found it extremely helpful because it explained what I was trying to say. I hope it can help others as well.

  3. Thank you for posting this. I really enjoyed reading it, especially because it addressed my question. It helped me a lot and I hope it will help others too.

  4. You actually make it seem so easy with your presentation but I find this topic to be actually something that I think I would never understand. It seems too complicated and very broad for me. I’m looking forward for your next post, I?ll try to get the hang of it!

  5. Excellent blog right here! Also your web site quite a bit up very fast! What host are you the usage of? Can I get your affiliate hyperlink to your host? I wish my website loaded up as fast as yours lol

  6. Great post. I was checking constantly this blog and I’m impressed! Extremely helpful information particularly the last part 🙂 I care for such information much. I was looking for this particular info for a long time. Thank you and good luck.

  7. As I site possessor I believe the content matter here is rattling fantastic , appreciate it for your hard work. You should keep it up forever! Best of luck.

  8. It?s really a great and helpful piece of info. I?m glad that you shared this useful info with us. Please keep us up to date like this. Thanks for sharing.

  9. whoah this blog is fantastic i love studying your posts. Stay up the great work! You already know, a lot of individuals are hunting around for this info, you can help them greatly.

  10. Hello my friend! I want to say that this post is amazing, nice written and include almost all vital infos. I?d like to see more posts like this.

  11. Very nice post. I just stumbled upon your blog and wanted to say that I’ve really enjoyed browsing your blog posts. In any case I?ll be subscribing to your feed and I hope you write again soon!

  12. Hi there, I discovered your site by means of Google at the same time as searching for a similar topic, your web site came up, it seems to be great. I’ve bookmarked it in my google bookmarks.

Leave a Reply

Your email address will not be published. Required fields are marked *