Enabling High-Speed Edge AI Processing and Contributing to Building of Optimal Systems Linked with the Cloud

Sony’s Intelligent Vision Sensors IMX500 (left) and IMX501 (right)

 

Sony Corporation today announced the upcoming release of two models of intelligent vision sensors, the first image sensors in the world to be equipped with AI processing functionality*1. Including AI processing functionality on the image sensor itself enables high-speed edge AI processing and extraction of only the necessary data, which, when using cloud services, reduces data transmission latency, addresses privacy concerns, and reduces power consumption and communication costs.

*1 Among image sensors. According to Sony research (as of announcement on May 14, 2020).

These products expand the opportunities to develop AI-equipped cameras, enabling a diverse range of applications in the retail and industrial equipment industries and contributing to building optimal systems that link with the cloud.

Sony's Intelligent Vision Sensors IMX500 (left) and IMX501 (right)

Sony’s Intelligent Vision Sensors IMX500 (left) and IMX501 (right)

Model name Sample shipment date Sample price (excluding tax)
IMX500 1/2.3-type (7.857 mm diagonal) approx. 12.3 effective megapixel intelligent vision sensor (bare chip product) April 2020 10,000 JPY
IMX501 1/2.3-type (7.857 mm diagonal) approx. 12.3 effective megapixel intelligent vision sensor (package product) June 2020 (planned) 20,000 JPY

The spread of IoT has resulted in all types of devices being connected to the cloud, making commonplace the use of information processing systems where information obtained from such devices is processed via AI on the cloud. On the other hand, the increasing volume of information handled in the cloud poses various problems: increased data transmission latency hindering real-time information processing; security concerns from users associated with storing personally identifiable data in the cloud; and other issues such as the increased power consumption and communication costs cloud services entail.

The new sensor products feature a stacked configuration consisting of a pixel chip and logic chip. They are the world’s first image sensor to be equipped with AI image analysis and processing functionality on the logic chip. The signal acquired by the pixel chip is processed via AI on the sensor, eliminating the need for high-performance processors or external memory, enabling the development of edge AI systems. The sensor outputs metadata (semantic information belonging to image data) instead of image information, making for reduced data volume and addressing privacy concerns. Moreover, the AI capability makes it possible to deliver diverse functionality for versatile applications, such as real-time object tracking with high-speed AI processing. Different AI models can also be chosen by rewriting internal memory in accordance with user requirements or the conditions of the location where the system is being used.

Main Features

■ World’s first image sensor equipped with AI processing functionality

The pixel chip is back-illuminated and has approximately 12.3 effective megapixels for capturing information across a wide angle of view. In addition to the conventional image sensor operation circuit, the logic chip is equipped with Sony’s original DSP (Digital Signal Processor) dedicated to AI signal processing, and memory for the AI model. This configuration eliminates the need for high-performance processors or external memory, making it ideal for edge AI systems.

■ Metadata output

Signals acquired by the pixel chip are run through an ISP (Image Signal Processor) and AI processing is done in the process stage on the logic chip, and the extracted information is output as metadata, reducing the amount of data handled. Ensuring that image information is not output helps to reduce security risks and address privacy concerns. In addition to the image recorded by the conventional image sensor, users can select the data output format according to their needs and uses, including ISP format output images (YUV/RGB) and ROI (Region of Interest) specific area extract images.

■ High-speed AI processing

When a video is recorded using a conventional image sensor, it is necessary to send data for each individual output image frame for AI processing, resulting in increased data transmission and making it difficult to deliver real-time performance. The new sensor products from Sony perform ISP processing and high-speed AI processing (3.1 milliseconds processing for MobileNet V1*2) on the logic chip, completing the entire process in a single video frame. This design makes it possible to deliver high-precision, real-time tracking of objects while recording video.

*2 MobileNet V1: An image analysis AI model for object recognition on mobile devices.

■ Selectable AI model

Users can write the AI models of their choice to the embedded memory and can rewrite and update it according to its requirements or the conditions of the location where the system is being used. For example, when multiple cameras employing this product are installed in a retail location, a single type of camera can be used with versatility across different locations, circumstances, times, or purposes. When installed at the entrance to the facility it can be used to count the number of visitors entering the facility; when installed on the shelf of a store it can be used to detect stock shortages; when on the ceiling it can be used for heat mapping store visitors (detecting locations where many people gather), and the like. Furthermore, the AI model in a given camera can be rewritten from one used to detect heat maps to one for identifying consumer behavior, and so on.

Key Specifications

Model name IMX500 (bare chip product) IMX501 (package product)
Number of effective pixels 4056 (H) × 3040 (V), approx. 12.3 megapixels
Image size Diagonal 7.857 mm (1/2.3 type)
Unit cell size 1.55 μm (H) × 1.55 μm (V)
Frame rate Full pixel 60 fps
Video 4K (4056 × 2288) 60 fps
1080p 240 fps
Full/video+AI processing 30fps
Metadata output 30fps
Sensitivity (F5.6 standard value) Approx. 250LSB
Sensor saturation signal level (minimum value) Approx. 9610e-
Power supply Analog 2.7V
Digital 0.84V
Interface 1.8V
Main functions AI processing function, ISP, HDR shooting
Output MIPI D-PHY 1.2 (4 lane) / SPI
Color filter array Bayer array
Output format Image (Bayer RAW), ISP output (YUV/RGB), ROI, metadata
Package IMX500: – IMX501: Ceramic LGA 12.5 mm (H) × 15.0 mm (V)

About Sony Corporation

Sony Corporation is a creative entertainment company with a solid foundation of technology. From game and network services to music, pictures, electronics, semiconductors and financial services – Sony’s purpose is to fill the world with emotion through the power of creativity and technology. For more information, visit: http://www.sony.net/

Sony Xperia 1 II  has made its appearance and is already an official phone with a surname that rings a bell. Sony tells us that it is pronounced “Xperia 1 Mark II”, welcome to the photographic line.

Sony Xperia 1 II – 5G mobile with camera and 4K OLED screen

Sony Xperia 1 II - 5G mobile with camera and 4K OLED screen

Sony Xperia 1 II – 5G mobile with camera and 4K OLED screen

And also as expected, Sony repeats with formulas that already worked for it last year and updates others, such as the logical leap in power brought about by new components such as the fashionable brain at this start of the year, the Qualcomm Snapdragon 865, which now he has more memory and more space as companions. And also new connectivity because yes,  the Sony Xperia 1 II gets on the 5G train . But there is more to scratch, and let’s see what has changed in this year between the two generations.

Adapting the screen to ‘gaming’ Sony style

Last year, Sony presented at MWC 2019 in Barcelona with a phone with a large screen. Not only big, but also very long. Thus, the  Sony Xperia 1  released an OLED panel  of nothing more and nothing less than 6.5 inches and with a 21: 9 ratio. CinemaDisplay, the manufacturer called it, and it was logical that the formula repeated this year with some change focused on the gamer public.

Sony even maintains 4K resolution for its new Xperia 1 II, although it takes a step forward in terms of refresh rate for its style. We do not have a 90Hz panel like the ones that are becoming fashionable, but  Sony chooses to mount the Motion Blur Reduction technology . With it, Sony interpolates frames using the AI ​​of the Snapdragon 865, thus emulating a higher refresh rate and improving the sharpness of its screen.

In the rest, everything is identical. We continue with 21: 9, with the protection provided by  Gorilla Glass in its sixth generation  and with the compatibility with HDR content, in addition to DCI-P3. As we see, the evolution in one year of Sony in terms of its screen has focused on adapting its panel to the ‘gaming’ trend, but we found many similarities between both generations. Potentially, however, things change.

Sony is stepping up photography efforts on its new mobiles to please five user profiles: cinematographers, photographers, moviegoers, audiophiles, and game lovers on mobile devices.

To do this, the 12-megapixel triple camera setup is now accompanied by a wider 24mm main sensor (vs. 26mm) with an f / 1.7 aperture and optical image stabilization, a further 70mm telephoto sensor long (also with OIS) and there is an ultra wide 16mm once again.

The main hook with this new sensor setup is the phone’s ultra-fast autofocus, which the Xperia team developed in collaboration with the Tokyo-based Alpha 9 camera team.

The focus area has been expanded to 70% of the frame and the camera’s intelligence is now also extended to make this the first autofocus phone that is capable of shooting up to 20 photos per second in burst mode with auto exposure.

The company’s real-time Eye-AF tracking, which previously tracked human eyes / faces, now works on pets, too. And to help the entire system perform these impressive photographic feats, Sony adds a new iToF 3D (the ‘i’ is short for AI-assisted depth detection).

Like the Cinema Pro app introduced last year (which now offers more control over frame rates, white balance, and integrates a smart wind filter), the Xperia 1 II now also includes a Photo Pro app.

  • Enjoy the renowned high-end sound of the 1000X family with the latest fully wireless headphones from Sony.
  • The next level of silence with Dual Noise Sensor technology and the HD Noise Canceling QN1e processor.
  • Wireless design, stable connectivity and comfortable fit for total listening freedom.
  • Long-lasting battery and quick charge function to revolutionize your daily commute to work.
  • Listen your way with Adaptive Sound Control, customizable touch controls and voice assistant.

The Sony WF-1000XM3 headphones

Sony today expands its renowned 1000X range with the addition of next-generation fully wireless headphones, the  WF-1000XM3 . The noise cancellation Sony, industry leader in January , has improved thanks to the processing chip HD Noise Canceling QN1e and Dual Noise Sensor technology, which delivers performance cancellation exceptional noise you can hear the music and nothing but the music.

The next level of silence

The WF-1000XM3 headphones feature Dual Noise Sensor technology that works with the new HD Noise Canceling QN1e processor to capture and cancel noise, respectively, so all the attention is focused on your music. Thanks to Dual Noise Sensor technology, the Feedforward and Feedback microphones on the headphone surface better capture the ambient sound around you. Once the external sound has been captured, the HD Noise Canceling QN1e processor gets to work. The processor intelligently creates an inverted sound wave that compensates for annoying background noise. Not only does it cancel out more noise on almost all frequencies, it also consumes less power. The headphones cancel out all kinds of noises, from the annoying noise of an airplane cabin to the hustle and bustle of the city streets,

  • Enjoy the renowned high-end sound of the 1000X family with the latest fully wireless headphones from Sony.
  • The next level of silence with Dual Noise Sensor technology and the HD Noise Canceling QN1e processor.
  • Wireless design, stable connectivity and comfortable fit for total listening freedom.
  • Long-lasting battery and quick charge function to revolutionize your daily commute to work.
  • Listen your way with Adaptive Sound Control, customizable touch controls and voice assistant.

Sony today expands its renowned 1000X range with the addition of next-generation fully wireless headphones, the  WF-1000XM3 . The noise cancellation Sony, industry leader in January , has improved thanks to the processing chip HD Noise Canceling QN1e and Dual Noise Sensor technology, which delivers performance cancellation exceptional noise you can hear the music and nothing but the music.

The next level of silence

The WF-1000XM3 headphones feature Dual Noise Sensor technology that works with the new HD Noise Canceling QN1e processor to capture and cancel noise, respectively, so all the attention is focused on your music. Thanks to Dual Noise Sensor technology, the Feedforward and Feedback microphones on the headphone surface better capture the ambient sound around you. Once the external sound has been captured, the HD Noise Canceling QN1e processor gets to work. The processor intelligently creates an inverted sound wave that compensates for annoying background noise. Not only does it cancel out more noise on almost all frequencies, it also consumes less power. The headphones cancel out all kinds of noises, from the annoying noise of an airplane cabin to the hustle and bustle of the city streets,

The WF-1000XM3 headphones are specially designed to improve comfort and stability in your ears. The “ergonomic triple grip” structure keeps the headphones firmly in place, with three support points for the ear. A high friction rubber surface has also been included to ensure a stable fit. With 4 sizes of hybrid earbuds and three sizes of triple comfort earbuds, you will always find the perfect fit.

Smart convenience

The HD Noise Canceling QN1e processor consumes little power, greatly increasing battery life and listening time. With a autonomy of up to 24 hours [1]  with the Noise Canceling function activated using the elegant earphone case WF-1000XM3 that works as a charger. This is perfect for more battery power when you’re away from home. The headphones have a quick charge function: just 10 minutes of charging offers up to 90 minutes of battery life.

For added ease, you can use Quick Attention mode to communicate without removing your headphones. Simply place your finger on the left earphone touchpad to quickly turn down the volume of the music and let the ambient sound in, so you can listen to announcements or order a drink, for example. Plus, with the headset’s fully customizable touch sensors [2] , you can easily play, pause, skip songs, or easily activate your voice assistant with a few simple taps. Plus, the new Wearing Detection feature automatically pauses music when you remove one of your headphones from your ear, and playback resumes when you put the headset back on, so you never miss a beat.

Optimized for the Google Assistant, you can control your headphones by voice, making the WF-1000XM3 the perfect companion for any trip [3] . The headphones are also compatible with the Sony application | Headphones Connect, where you will find sound customization functions, noise cancellation, touch controls, among many others.

The WF-1000XM3 model will cost approximately € 250 and will be available from the beginning of August.

For product specifications, visit the  Sony website .

Up to 24 hours with the Noise Canceling function activated (6 hx 3 charges) and up to 32 hours with the Noise Canceling function deactivated (8 hx 3 charges). The case provides up to 3 additional charges.

Customizable through the Sony application | Headphones Connect.

Google is a registered trademark of Google LLC.

About Sony Corporation

Sony Corporation is a creative entertainment company with a strong technology base. Through its divisions, ranging from video games and online services, through music, film, electronics, semiconductors and financial services, Sony’s goal is to feed the illusion thanks to the power of creativity and technology. More information can be found on Sony’s global website:  http://www.sony.net/