Blogs/Understanding Derivatives

Understanding Derivatives

peterwashington Nov 03 2021 19 min read 0 views

The fundamental building block of calculus is the derivative. The derivative is a way of measuring the rate of change of an equation as one of the variables in that equation changes. To understand the derivative conceptually, let’s look at the graph for the function y= x2+1:

Different points of the function will have different rates of change. We are interested in how y changes with respect to changes in x. Let’s consider the 3 points A, B, and C above:

  • Point A is at x= -1.75. At this point, the function is decreasing, so it will have a negative rate of change at x= -1.75. The rate of decrease is relatively high at this point. The rate of change is -3.5.
  • Point B is at x= 0. Because the function is neither increasing nor decreasing at this point, the derivative is 0.
  • Point C is at x= 1. At this point, the function is increasing, so it will have a positive rate of change at x= 1. Because point C is increasing more slowly / less steeply than point A was decreasing, point C will have a smaller magnitude (absolute value) of its rate of change. The rate of change at point C is 2.

The rate of change, or derivative, of a particular point of a function is the slope of the line which is tangent to that point on the curve. Here is a visualization of the tangent lines for points A, B, and C:

We can see that the slope of these tangent lines represents the rate of change of the curve at that point.

Calculus provides a toolkit for calculating derivatives from functions.

The math notation for a derivative is dy/dx. This notation means taking the derivative of the function y with respect to x. So, we could write that:

The last thing you need to understand about derivatives is that the variable you are taking the derivative with respect to matters. Let’s say we have the function f(x,y,z)= 3x5-12xy+2z-5

Then, the following 3 derivatives differ based on whether we are taking the derivative with respect to x, y, or z:

When multiple variables are involved and we only take the derivative with respect to one particular variable, like in the example above, then we call this a partial derivative. Partial derivatives are used in gradient descent, as we are taking the derivative with respect to each of the weights of a model.

The general formula for a derivative is the following, where we are finding the slope of tangent line at x=a and y=f(a):

Let’s break down this formula to understand it. First, we need to understand the concept of a limit. The lim operator in math means that we find what the value of f(x) approaches as x gets increasingly closer to A, or “the limit of f(x) as x approaches A”.

For example, imagine the following function:

When x = -1, then the above equation is undefined, since we cannot divide by 0. However, notice that as x gets increasingly closer to -1, the value of f(x) gets increasingly closer to -2:

Therefore, we can say that the limit is -2.

The formula for a derivative involves finding the slope of the tangent line at a point x=a and y=f(a). In order to approximate this tangent line, we can imagine finding the slope of a line going from point (a, f(a)) to a point very close to this point (a + h, f(a + h)):

This is called a secant line. As the distance from our point of interest, h, gets closer to the point of interest (h gets smaller), we get an increasingly better approximation of the slope of the tangent line at our point of interest:

Eventually, h is so small that we have the tangent line:

Let’s again look at the general formula for a derivative, with all of this in mind:

Intuitively, we can think of this as the rise over run of the secant line going from (a, f(a)) to (a + h, f(a + h)). As h gets increasingly closer to 0, this slope of the secant line becomes increasingly closer to the slope of the tangent line at (a, f(a)). And that’s how we get our general formula for the derivative!