Delivering products with care, honoring commitments to clients
In the last article “A Race Against Time: Launching Livox HAP Onto the Roads”, we explored how the Livox teams overcame hurdles in creating the production line for and the manufacturing of mass-production automotive-grade LiDARs. At Livox, we are constantly striving to fine-tune our project delivery management system, guided by the belief that our success hinges on that of our customers. Our first mass-production LiDAR project has been a great opportunity for the Livox teams to hone their skills and improve their processes.
To the Livox teams, managing the production of LiDARs is like a marathon: As the demand for mass-production LiDARs grows, to simplify and streamline processes is a bigger challenge than to deal with increased workloads. With that in mind, Livox chose to focus its resources on a mass-production project with great significance for the industry, where it could concentrate on trialing and refining various systems and processes, to prepare itself for the industry’s widespread adoption of the production model in the future.
During this process, the team also learned that a successful product delivery depends on the cooperation of multiple departments, such as project management, R&D, procurement, manufacturing, quality assurance, sales, aftersales services, and finance. In order to ensure supply continuity during the chip shortage, some teams verified more than 60 alternative materials in as short a period as five months. Some designed more than 20 different hardware solutions, and made sufficient redundant backups just in case they are needed. On an occasion, some staff had to fly to the supplier’s plant to collect materials directly, for a customer’s production schedule slated for 7 am the next day. To meet the nine-month delivery period for the project, a few teams expedited the reliability design and verification processes for closed-loop products by running over a thousand samples beyond the DV test standards.
All of this was driven by our belief that what Livox delivers is more than products––––it is a promise. It is only by becoming a better “we” throughout this journey, that we can deliver the best to our customers.
01.Going the extra Mile to Safeguard Delivery
Since 2021, the world has been grappling with an unprecedented chip shortage crisis, with the delivery cycle for some materials even extended to 18 months or longer. It is often the case where a delivery deadline is looming while the supply chain is dealing with all kinds of disruption in the upstream. Manufacturers are always chasing after materials, or on their way to do so.
Keenly aware of the situation, the Livox procurement team quickly gathered their colleagues at R&D to find solutions to the chip shortage. How to ensure consistent delivery when production demand is growing by tenfold, the chip supply chain is disrupted, and the hardware designs have all been fixed? That was the most pressing issue facing the hardware team leader Han Ruiyuan and his team members.
Ruiyuan recalled: “Constrained by the fixed hardware designs and the high-density chip layout, any changes we made to any material would affect everything. Many areas of the operation would be greatly impacted, such as the hardware system metrics, EMC test, reliability verification, workstation ICT/FCT test, software compatibility, and subsequent procurement. An even bigger hurdle was that, if supply was disrupted for multiple chips, we would end up with permutations involving many developed versions and their verifications. This would be the ultimate challenge for our hardware team, in terms of their management, competence, and resilience.”
Typically, the high-performance and high-reliability designs of automotive electronic products are supported by a sophisticated system. In the case of LiDARs, however, developers cannot fully depend on traditional systems to strike a balance between the two aspects. At the beginning of the project, Ruiyuan already recognized that hardware development for the first automotive-grade LiDAR posed a brand-new question to the whole hardware team: What basic rules and standards should be followed for hardware design in the unchartered territory of automotive-grade LiDARs?
On the component level, only automotive-grade electronic components can be used on the LiDARs, meaning they must be AEC-Q certified. Their design process also has to meet automotive electronic design and development requirements, such as WCA (Worst Case Analysis).
Take the switching power supply feedback resistor as an example, its resistance in an industrial-grade product only needs to fulfill the resistance value, precision, and power requirements within a certain range. However, in a WCA of automotive-grade product, the developer has to analyze multiple performance metrics of the component’s resistance, such as its precision temperature drift within the full temperature range, its resistance changes within the full humidity range, and its precision drift within its life cycle. Meanwhile, during the design process, engineers are subject to stringent development requirements. Besides calculating the boundaries of the output voltage of the switching power supply under different temperatures, humidity and stress scenarios, they also have to be constantly mindful of its risk of failure.
Moving from components to product development, a V model is generally followed for the latter phase. However, hardware design only accounts for the left half of the model while the right half also involves performance testing and product verifications.
As a photoelectronic product, a LiDAR’s performance is determined directly by its electrical design. To ensure design quality, implementing the V model requires the testing of multiple metrics such as the signals and power integrity at the board level, function and performance at the module and vehicle levels, EMC, and electrical/ environmental/ mechanical reliability. Therefore, even a small design change could trigger changes and additions to a series of subsequent tests and verifications. The complexity and workload involved in the hardware design for automotive-grade products far exceed those for common industrial-grade products.
V Model for Automotive Electronic Product Development
By the end of June 2021, the chip supply chain pressure quickly shifted from the procurement to the R&D team. To solve this problem, Ruiyuan said they “evaluated and switched to over 60 alternative materials and made more than 20 changes to the PCBA board design in just five months.”
For example, in evaluating and selecting a new automotive-grade switching power chip, they were not only constrained by the limited range of automotive-grade chips and its small layout, but also by the component’s input and output voltage and its precision, conversion efficiency, ripple size, EMI, reliability and several other parameters. At the same time, to ensure the product’s high reliability throughout its life cycle, the automotive-grade component must also pass a number of tests such as for AEC-Q certification and computing device FIT verification. As such, the entire process is cumbersome and time-consuming, and makes the evaluation of new components complicated and difficult.
“We went through over 20 versions for the PCBA board design, and throughout the process, we did not encounter any design error or quality incident. This was incredible given the high pressure and fast pace of work we were dealing with. This is not only a reflection of great teamwork, but also provides an opportunity for the hardware team to elevate its design skills for HAP projects,”Ruiyuan said.
To overcome the chip shortage and ensure product delivery, the team made over 20 design revisions to the hardware to come up with multiple alternative solutions.
02.On Time and in Full
In the context of delivery, a common term used in the supply chain industry is “OTIF”, which stands for “on time and in full”. Basically, it is a metric for measuring the degree to which a product is delivered to a customer in a timely manner, at the agreed quality, and in the agreed quantity.
Given it was the industry’s first mass production project for high-performance automotive-grade LiDARs, all teams in every production segment were dealing with unprecedented challenges. Some teams were in charge of timeliness of delivery, while some were responsible for delivering samples or mass production items in quantities required by customers. For certain teams, their job was to ensure the quality of delivered products.
A product’s quality is not only reflected in the stable and reliable performance of hardware, but also whether its software is viable upon delivery.
Su Wei was the manager in charge of product reliability testing. With little to no precedents to follow, she was faced with the demanding task of pacing their DV tests with every rapidly updated design version.
For regular projects, a single-round test cycle usually takes about three to four months. It was obvious that a conventional design for the DV tests would not be able to meet the mass production schedule required by the client. After multiple discussions, the team finally adopted a design scheme that involved rapid updates, multiple test rounds, and simultaneous verification, to achieve full coverage of all product design verifications and strict control of reliability-related design elements, and to significantly shorten the product update cycle.
Livox HAP’s reliability laboratory
For reliability tests, the customer’s acceptance criteria was for at least six samples to meet the test requirements in each sample group (with 10 groups in total). Theoretically, the team only had to include 60 samples in one round of testing. However, to ensure the high performance and reliability of the first automotive-grade LiDAR, the project team tested nearly 1,000 samples for reliability throughout the entire project development cycle, under highly rigorous conditions. During the process, the durability test alone was conducted on more than 500 samples just to speed up the simulation of a years-long service life for the LiDAR in regular driving scenarios. Su Wei acknowledged that: “Vehicle-mounted LiDARs are a new area with new challenges for the industry. We are lucky enough to be the first to test the waters, even if that means investing several times more resources to find smarter ways to get things done in a more stringent environment. Last year was already tough in terms of on-time and in-full delivery, but to me and my team, who are responsible for product quality, the ‘full” in ‘on-time and in-full’ was not just about quantity, but also quality.”
In addition to hardware reliability, the delivery requirements for automotive-grade software are equally strict. As a new-energy vehicle maker, XPeng together with Livox are on the cutting edge of automotive-grade LiDARs. In this shared mission to achieve breakthroughs, neither lowered their requirements but instead went over and above to make sure every production detail conformed to the highest standards.
Tang Yunfan, head of software development, also shared his experience working on the project: “We had to release rapid updates for the automotive-grade software within a short period of time, while making sure they are 100% stable and reliable in quality. That means stricter and more extreme requirements for delivery. For example, the software release cycle was two to three months, but when multiple test verified versions were required, we eventually had to release a new version every one to two weeks. More importantly, we must ensure the integrity of the software design in accordance with industry standards. Throughout the project, we released more than 20 software versions to our customer. For that, we created more than 400 updated internal versions.”
Take the customer’s strict requirements for Over-the-air (OTA) Technology features as an example. For each released software version, the client performed a bench upgrade test for at least another 100 hours and a full-vehicle OTA stress test for 400 hours. The software would only be accepted if no upgrade failure had occurred during the tests. To meet these stringent requirements, the team followed the ASPICE standards to ensure software quality across multiple aspects, from design, implementation to testing. For example, during the OTA design phase, the team successfully identified a number of scenarios such as sudden power failure or network disconnection, flash write or erase failures, and compatibility with multiple hardware versions (due to hardware shortage) through FMEA. To avoid these scenarios, multiple mechanisms such as loader backup, program integrity check, and security verification were added to ensure the security and stability of the software. In addition, before each firmware release, the team would first perform a stress test, software self-test, reliability test, simulated customer bench test, and other multi-level OTA upgrade tests on the application and loader. These involved a total of 112 test cases, of which more than 2,000 single-round tests were conducted and each took more than one week to complete.
Besides updates, debugging was also an essential day-to-day task to guarantee the software’s function. The most memorable episode for Yunfan was when they had to deal with an upgrade-jamming bug that occurred suddenly with a software version. Even though it was an inactive phantom bug, the team did not want to leave anything to chance. Not ones to give up easily, they pored over every module and technical manual, before successfully pinning down the root cause after a month. Thereafter, the team performed 60,000 cold-start and hot-start tests as well as bench upgrade verifications for two weeks, to ensure the error had been solved fundamentally.
It has been 12 months since the project’s conclusion, and XPeng vehicles with their NGP system supported by Livox HAP can be seen traveling smoothly and safely on many urban roads. At the same time, we are glad to have found that the test standards set by Su Wei and her team one year ago are highly consistent with the international LiDAR standards currently implemented by the industry. “Roads are made by people.” That’s the spirit of the Livox teams, along with their pragmatism, passion, and dedication.
03.The 24-hour Trip For a Chip
It is well-known that a general delivery cycle for chip materials is about 8 to 12 months. But with the supply chain disruptions last year, this time frame has been extended to over 18 months. Due to the customer’s increased demand and their requirement of a less-than-five-month delivery cycle, chip delivery has become an extremely critical and time-sensitive issue for the team.
When sharing his experience in chasing after materials during that period, Chen Ke, a colleague in charge of the supply chain, was able to specify the dates of the events. “At the Shanghai Auto Show on April 19, we invited the senior managers of more than 20 semiconductor suppliers to a discussion. We took the opportunity to explain to them the progress of our mass-production project, our follow-up plan, and the industry outlook for the product, hoping to secure the support of these upstream supply chain partners for our first automotive-grade LiDARs. The session not only provided our suppliers with a deeper understanding of our operations, but also laid a good foundation for the smooth delivery of the project later.” In May and June, the procurement team quickly locked in the delivery dates with the various suppliers.
Little did we expect our plans would soon be derailed. Despite the team’s early forecast of the supply chain situation, the chip shortage still turned out to be more severe than expected. In June, the suppliers began alerting us of potential risks of supply delays. To cope with the new risks and show our trust in our suppliers, the procurement team mobilized all its resources and even facilitated a three-party meeting between Livox, our customer, and the suppliers. To that end, the procurement leader visited the suppliers’ Chinese headquarters every month, and the team also set up a system of routine communication with the senior management at their overseas headquarters. After dozens of hours of meetings across time zones, the procurement team was finally able to work out a solution with the suppliers to fulfill our most pressing needs.
During our interview, we asked the project manager Du Kai about his most memorable event from the delivery process. After pondering briefly, he said: “It would be the ‘24-hour journey for a chip’.”
“Late September last year, we already started delivering in small batches. At the time, we urgently needed a material so that production could commence as scheduled. We knew that our customer calculated their quantity of vehicles for sale by the minute, so even a minute of delay could affect their production timeline. Even with express courier, the materials wouldn’t make it to us in time for the customer. Therefore, we decided to have our colleagues take the earliest flight that day to the supplier’s location and collect the materials from their factory, then fly back on the same afternoon. As fate would have it, there was a storm on their way back and their flight was delayed a few times. In the end, they made it back to Shenzhen at 2:00 am the following day. During this period, we coordinated the colleagues at the warehouse to rejig their inbound logistics plan so that personnel would be on hand to inspect and accept materials received outside of regular working hours and have them ready for production. Eventually, the LiDARs were shipped “fresh out of the oven” from Shenzhen at dawn, and arrived before the start of the customer’s morning shift at their factory.”
To ensure the customer’s production could commence on time, the procurement team was constantly checking with the suppliers on their delivery dates while also making sure every batch of materials was safely received at the warehouse. Their success relied on not just their keen sensitivity to supply chain risks, but also the dedication of every colleague at every step of the operation. They were assertive when making decisions against new variables, and persevered when overcoming a hurdle. In the end, they made the project a success with their conviction and determination.
After the customer’s production had entered the standardized stage, their sales team told us: “I miss the time when we had to synchronize data every two hours and check receipts of goods until early morning, in this whole experience of transitioning from our largely manual pilot production line to the fully automated mass production line, and devising all kinds of backup plans.”
At Livox, we are well aware that behind every project’s success is the concerted effort of every person involved. We have always believed that what Livox delivers is more than products. It is a promise. With every successful delivery, we strengthen further the mutual trust we have with our partners.
Posted on 2022-9-28 10:59:59