Introduction

If you are a software programmer then you know the pain of debugging production issues. But if you don't know let me explain what debugging process is:

Debugging is a process of identifying potential bugs or errors in programming code. It is an essential part of software programming as it helps software developers to identify and resolve issues in their programs that do not produce the desired or expected output for a given set of inputs.

Still not clear what debugging is? Then let me walk you around one of my debugging adventures as a backend engineer.

Adventure Begins

In production for the backend, we are using PHP for powering the mobile app and website of our company. There is one function that is responsible to give all data related to order based on orderId. As this function is so complex in nature it executes many database queries for fetching all data related to orderId even if a client needs a subset of data. So we decided to break this function into multiple functions with each function will give a subset of data of the original function since most of the client needs a certain subset of the order data.

For example, think as we have a single function upon giving an orderId it provides details like order rating, customer details who ordered, dishes ordered etc. We tried to divide this function into multiple functions like a function upon giving an orderId will give only order rating, customer details, or dishes details.

func F1(orderId)
return (
orderRating,
orderDishes,
customerDetails,
orderState,
paymentState,
)

above function will break into the following functions:
func F2(orderId) return orderRating
func F3(orderId) return orderDishes
func F4(orderId) return customerDetails
func F5(orderId) return orderState
func F6(orderId) return paymemtState

In order to know which client needs which subset of data we decided to add debug backtrace on the former function.

As the code was in PHP, we used debug_backtrace() inbuilt function of PHP to get the backtrace of each calling function.

Adventure To Disaster

As soon as we take this code live. All the workers which were calling the former functions were getting crashed because of out of memory issues. Soon we realized there was a potential bug in our code. As our worker used to run in an infinite loop to consume messages from the queue this debug_backtrace() function was taking a lot of memory and processing power for each next iteration of the loop.
We ran below test function locally
func Test() {
while(true) {
debug_backtrace();
}
}

The above function was also consuming a lot of memory and processing power for each next iteration of the loop and hence we removed the debug_backtrace() function.

Learning from Adventure

We now always deprive ourselves to use debug_backtrace() or any other debugging function for production usecases. These debug functions generally have a negative impact on the performance and security of the system and can lead to a major outrage in production.

Conclusion

I now hope you know what is debugging and how we can fix the potential bug in software programming code.

- Debugging is an art not a science :)