Say Goodbye to Lag: Generate Lightning-Fast Image Blurhash for Flutter Web!
How to process ImageData in Uint8List faster using HTML
In this blog, we will learn how to process Uint8List ImageData faster. We will use the example of Blur-hash generation. Blur-hash is the placeholder of the image while it is loading.
The Blur-hash algorithm creates a blurred, low-resolution version of an image that can be encoded into a short string. This string is then decoded on the client side to reconstruct the blurred image. You can read more here: 🔗 https://blurha.sh/
To generate a Blur-hash, we use an algorithm which takes Image object in dart and generate a hash string.
The problem
Let's generate a Blur-hash(BH) of an image. We will use a plugin to generate the BH : 🔗 https://pub.dev/packages/blurhash
import 'dart:typed_data';
import 'package:image/image.dart' as img;
import 'package:blurhash_dart/blurhash_dart.dart';
Future<String> getBlurHashFromUint8list(Uint8List imageData) async {
final image = img.decodeImage(imageData);
final blurHash = BlurHash.encode(image!, numCompX: 4, numCompY: 3);
return blurHash.hash;
}The above method will generate the BH it is straight forward. Let's see this is action.
The above video shows image BH generation lags our app and our app become unresponsive while BH is generating. We are using
decodeImage()to generate the BH.
The above method generates the BH but our app lags while the BH is being generated. The decodeImage() is very slow process, it convert image bytes into single frame image.
In most of the image manipulation task we will need to generate ImageData object. In Flutter web the decodeImage() is very slow we need to optimize this method to make our image processing fatser.
The Solution
To solve this we will use native javascript method to generate ImageData object.
import 'dart:async';
import 'dart:html';
import 'dart:typed_data';
import 'dart:convert';
import 'package:flutter/foundation.dart';
import "package:universal_html/html.dart" as html;
Future<ImageData?> getImageData(Uint8List image, String mimeType) async {
int width, height;
String jpg64 = base64Encode(image);
html.ImageElement myImageElement = html.ImageElement();
myImageElement.src = 'data:$mimeType;base64,$jpg64';
await myImageElement.onLoad.first;
if (myImageElement.width! > myImageElement.height!) {
width = 100;
height = (width * myImageElement.height! / myImageElement.width!).round();
} else {
height = 100;
width = (height * myImageElement.width! / myImageElement.height!).round();
}
html.CanvasElement myCanvas = html.CanvasElement(
width: width,
height: height,
);
html.CanvasRenderingContext2D ctx = myCanvas.context2D;
ctx.drawImageScaled(myImageElement, 0, 0, width, height);
html.ImageData imageData = ctx.getImageData(0, 0, width, height);
return imageData;
}
The above method will generate ImageData object using HTML ImageElement() method. We used HTML methods to generate the ImageData instead of Dart method as the HTML is incredibly faster for web platform.
Lets generate BH using the new ImageData we generated above.
import 'dart:async';
import 'dart:convert';
import 'dart:html';
import 'dart:js';
import 'package:flutter/foundation.dart';
import 'package:flutter_image_compress/flutter_image_compress.dart';
import "package:universal_html/html.dart" as html;
Future<String> getBlurHashFromUint8list(Uint8List uint8List) async {
var imageDataNatively = await getImageData(uint8List, 'image/jpeg');
final data = imageDataNatively?.data;
final width = imageDataNatively?.width;
final height = imageDataNatively?.height;
var blurhash = context.callMethod('encode', [data, width, height, 4, 3]);
return blurhash;
}
The BH generation is incredibly fast now! And our app does not become unresponsive. The above video shows BH genaration using HTML methods instead of dart.
We have also used native javascript Blurhash encoding instead of Flutter BH plugin as native js BH encoding is faster.
To generate BH using javascript we need to add blurhash_generator.js file with BH generation code.
Download the below blurhash_generator.js file from this link: https://gist.github.com/abdulrehmank7/25f0cf05284320b045461feb0547d784
const digitCharacters = ["0","1","2","3","4","5","6","7","8","9","A","B","C","D","E","F","G","H","I","J","K","L","M","N","O","P","Q","R","S","T","U","V","W","X","Y","Z","a","b","c","d","e","f","g","h","i","j","k","l","m","n","o","p","q","r","s","t","u","v","w","x","y","z","#","$","%","*","+",",","-",".",":",";","=","?","@","[","]","^","_","{","|","}","~"];
const decode83 = str => {
let value = 0;
for (let i = 0; i < str.length; i++) {
const c = str[i];
const digit = digitCharacters.indexOf(c);
value = value * 83 + digit;
}
return value;
};
const encode83 = (n, length) => {
var result = "";
for (let i = 1; i <= length; i++) {
let digit = (Math.floor(n) / Math.pow(83, length - i)) % 83;
result += digitCharacters[Math.floor(digit)];
}
return result;
};
const sRGBToLinear = value => {
let v = value / 255;
if (v <= 0.04045) {
return v / 12.92;
} else {
return Math.pow((v + 0.055) / 1.055, 2.4);
}
};
const linearTosRGB = value => {
let v = Math.max(0, Math.min(1, value));
if (v <= 0.0031308) {
return Math.round(v * 12.92 * 255 + 0.5);
} else {
return Math.round(
(1.055 * Math.pow(v, 1 / 2.4) - 0.055) * 255 + 0.5
);
}
};
const sign = n => (n < 0 ? -1 : 1);
const signPow = (val, exp) => sign(val) * Math.pow(Math.abs(val), exp);
const validateBlurhash = blurhash => {
if (!blurhash || blurhash.length < 6) {
throw new Error(
"The blurhash string must be at least 6 characters"
);
}
const sizeFlag = decode83(blurhash[0]);
const numY = Math.floor(sizeFlag / 9) + 1;
const numX = (sizeFlag % 9) + 1;
if (blurhash.length !== 4 + 2 * numX * numY) {
throw new Error(
`blurhash length mismatch: length is ${
blurhash.length
} but it should be ${4 + 2 * numX * numY}`
);
}
};
const isBlurhashValid = blurhash => {
try {
validateBlurhash(blurhash);
} catch (error) {
return { result: false, errorReason: error.message };
}
return { result: true };
};
const decodeDC = value => {
const intR = value >> 16;
const intG = (value >> 8) & 255;
const intB = value & 255;
return [sRGBToLinear(intR), sRGBToLinear(intG), sRGBToLinear(intB)];
};
const decodeAC = (value, maximumValue) => {
const quantR = Math.floor(value / (19 * 19));
const quantG = Math.floor(value / 19) % 19;
const quantB = value % 19;
const rgb = [
signPow((quantR - 9) / 9, 2.0) * maximumValue,
signPow((quantG - 9) / 9, 2.0) * maximumValue,
signPow((quantB - 9) / 9, 2.0) * maximumValue
];
return rgb;
};
const bytesPerPixel = 4;
const multiplyBasisFunction = (pixels, width, height, basisFunction) => {
let r = 0;
let g = 0;
let b = 0;
const bytesPerRow = width * bytesPerPixel;
for (let x = 0; x < width; x++) {
for (let y = 0; y < height; y++) {
const basis = basisFunction(x, y);
r +=
basis *
sRGBToLinear(
pixels[bytesPerPixel * x + 0 + y * bytesPerRow]
);
g +=
basis *
sRGBToLinear(
pixels[bytesPerPixel * x + 1 + y * bytesPerRow]
);
b +=
basis *
sRGBToLinear(
pixels[bytesPerPixel * x + 2 + y * bytesPerRow]
);
}
}
let scale = 1 / (width * height);
return [r * scale, g * scale, b * scale];
};
const encodeDC = value => {
const roundedR = linearTosRGB(value[0]);
const roundedG = linearTosRGB(value[1]);
const roundedB = linearTosRGB(value[2]);
return (roundedR << 16) + (roundedG << 8) + roundedB;
};
const encodeAC = (value, maximumValue) => {
let quantR = Math.floor(
Math.max(
0,
Math.min(
18,
Math.floor(signPow(value[0] / maximumValue, 0.5) * 9 + 9.5)
)
)
);
let quantG = Math.floor(
Math.max(
0,
Math.min(
18,
Math.floor(signPow(value[1] / maximumValue, 0.5) * 9 + 9.5)
)
)
);
let quantB = Math.floor(
Math.max(
0,
Math.min(
18,
Math.floor(signPow(value[2] / maximumValue, 0.5) * 9 + 9.5)
)
)
);
return quantR * 19 * 19 + quantG * 19 + quantB;
};
/**
* @param {Uint8ClampedArray} pixels
* @param {Number} width
* @param {Number} height
* @param {Number} componentX
* @param {Number} componentY
* @returns {Promise<String>}
*/
encodePromise = (pixels, width, height, componentX, componentY) => {
return new Promise((resolve, reject) => {
resolve(
encode(pixels, width, height, componentX, componentY)
);
});
};
/**
* @param {Uint8ClampedArray} pixels
* @param {Number} width
* @param {Number} height
* @param {Number} componentX
* @param {Number} componentY
* @returns {String}
*/
encode = (pixels, width, height, componentX, componentY) => {
if (
componentX < 1 ||
componentX > 9 ||
componentY < 1 ||
componentY > 9
) {
throw new Error("BlurHash must have between 1 and 9 components");
}
if (width * height * 4 !== pixels.length) {
throw new Error("Width and height must match the pixels array");
}
let factors = [];
for (let y = 0; y < componentY; y++) {
for (let x = 0; x < componentX; x++) {
const normalisation = x == 0 && y == 0 ? 1 : 2;
const factor = multiplyBasisFunction(
pixels,
width,
height,
(i, j) =>
normalisation *
Math.cos((Math.PI * x * i) / width) *
Math.cos((Math.PI * y * j) / height)
);
factors.push(factor);
}
}
const dc = factors[0];
const ac = factors.slice(1);
let hash = "";
let sizeFlag = componentX - 1 + (componentY - 1) * 9;
hash += encode83(sizeFlag, 1);
let maximumValue;
if (ac.length > 0) {
let actualMaximumValue = Math.max(
...ac.map(val => Math.max(...val))
);
let quantisedMaximumValue = Math.floor(
Math.max(
0,
Math.min(82, Math.floor(actualMaximumValue * 166 - 0.5))
)
);
maximumValue = (quantisedMaximumValue + 1) / 166;
hash += encode83(quantisedMaximumValue, 1);
} else {
maximumValue = 1;
hash += encode83(0, 1);
}
hash += encode83(encodeDC(dc), 4);
ac.forEach(factor => {
hash += encode83(encodeAC(factor, maximumValue), 2);
});
return hash;
};
// utils
/**
*
* @param {Image} img
* @returns {Uint8ClampedArray}
*/
getImageData = img => {
const width = img.width;
const height = img.height;
const canvas = document.createElement("canvas");
const ctx = canvas.getContext("2d");
canvas.width = width;
canvas.height = height;
ctx.width = width;
ctx.height = height;
ctx.drawImage(img, 0, 0);
return ctx.getImageData(0, 0, width, height).data;
};
/**
* @param {Image} img
* @param {Number} width
* @param {Number} height
* @returns {HTMLCanvasElement}
*/
drawImageDataOnNewCanvas = (imgData, width, height) => {
const canvas = document.createElement("canvas");
const ctx = canvas.getContext("2d");
canvas.width = width;
canvas.height = height;
ctx.width = width;
ctx.height = height;
ctx.putImageData(new ImageData(imgData, width, height), 0, 0);
return canvas;
};
/**
* @param {Uint8ClampedArray} imgData
* @param {Number} width
* @param {Number} height
* @returns {Promise<Image>}
*/
getImageDataAsImageWithOnloadPromise = (imgData, width, height) => {
return new Promise((resolve, reject) => {
getImageDataAsImage(
imgData,
width,
height,
(event, img) => {
resolve(img);
}
);
});
};
/**
* @param {Uint8ClampedArray} imgData
* @param {Number} width
* @param {Number} height
* @param {function} onload on image load
*/
getImageDataAsImage = (imgData, width, height, onload) => {
const canvas = drawImageDataOnNewCanvas(imgData, width, height);
const dataURL = canvas.toDataURL();
const img = new Image(width, height);
img.onload = event => onload(event, img);
img.width = width;
img.height = height;
img.src = dataURL;
return img;
};After downloading the js file, add the file in web directory of your project. And then add script tag in index.html
<head>
...
...
<!-- This script adds the flutter initialization JS code -->
<script src="flutter.js" defer></script>
<script src="blurhash_generator.js"></script>
</head>Now we can use the above javascript code in our Flutter app.
var blurhash = context.callMethod('encode', [data, width, height, 4, 3]);The above method will call the javascript encode method from our blurhash_generator.js file.
You can read more about js-interop here: https://dart.dev/interop/js-interop
Result
After updating the getBlurHashFromUint8list() to use native BH encoding we have significantly improved our Flutter web app. Their is no jank or lag when our app is generating BH.
🐢 Before unresponsive web app 🐢
⚡️New flow with native BH generation.⚡️
Full code
import 'dart:async';
import 'dart:convert';
import 'dart:html';
import 'dart:js';
import 'package:flutter/foundation.dart';
import 'package:flutter_image_compress/flutter_image_compress.dart';
import "package:universal_html/html.dart" as html;
Future<ImageData?> getImageData(Uint8List image, String mimeType) async {
int width, height;
String jpg64 = base64Encode(image);
html.ImageElement myImageElement = html.ImageElement();
myImageElement.src = 'data:$mimeType;base64,$jpg64';
await myImageElement.onLoad.first;
if (myImageElement.width! > myImageElement.height!) {
width = 100;
height = (width * myImageElement.height! / myImageElement.width!).round();
} else {
height = 100;
width = (height * myImageElement.width! / myImageElement.height!).round();
}
html.CanvasElement myCanvas = html.CanvasElement(
width: width,
height: height,
);
html.CanvasRenderingContext2D ctx = myCanvas.context2D;
ctx.drawImageScaled(myImageElement, 0, 0, width, height);
html.ImageData imageData = ctx.getImageData(0, 0, width, height);
return imageData;
}
Future<String> generateBlurhashFromImage(Uint8List uint8List) async {
var imageDataNatively = await getImageData(uint8List, 'image/jpeg');
final data = imageDataNatively?.data;
final width = imageDataNatively?.width;
final height = imageDataNatively?.height;
var blurhash = context.callMethod('encode', [data, width, height, 4, 3]);
return blurhash;
}Generate Blurhash of video in same way
import 'dart:async';
import 'dart:convert';
import 'dart:html';
import 'dart:js';
import 'package:flutter/foundation.dart';
import 'package:flutter_image_compress/flutter_image_compress.dart';
import "package:universal_html/html.dart" as html;
Future<Uint8List?> getImageFromVideo(Uint8List videoData) async {
Blob videoBlob = Blob([videoData], 'video/mp4');
String videoUrl = Url.createObjectUrlFromBlob(videoBlob);
html.VideoElement video = VideoElement()
..crossOrigin = 'anonymous'
..src = videoUrl
..muted = true;
CanvasElement canvas = CanvasElement();
video.onLoadedMetadata.listen((Event event) async {
canvas.width = video.videoWidth;
canvas.height = video.videoHeight;
video.currentTime = 0;
});
Completer<Uint8List?> completer = Completer();
video.onSeeked.listen((Event event) async {
CanvasRenderingContext2D ctx = canvas.context2D;
ctx.drawImage(video, 0, 0);
Blob blob = await canvas.toBlob('image/png', 0.95);
XFile xFile = XFile(
Url.createObjectUrlFromBlob(blob),
mimeType: 'image/png',
lastModified: DateTime.now(),
length: blob.size,
);
debugPrint('Obtained xFile=[${xFile.path}] for path=[$videoUrl].');
completer.complete(await xFile.readAsBytes());
});
video.onError.listen((Event event) async {
debugPrint('Error processing path=[$videoUrl] with event=[$event].');
completer.complete(null);
});
return completer.future;
}
Future<(String, Uint8List)> generateBlurhashFromVideo(
Uint8List uint8List) async {
final thumbnailData = await getImageFromVideo(uint8List);
var imageDataNatively = await getImageData(thumbnailData!, 'image/jpeg');
final data = imageDataNatively?.data;
final width = imageDataNatively?.width;
final height = imageDataNatively?.height;
var blurhash = context.callMethod('encode', [data, width, height, 4, 3]);
return (blurhash as String, thumbnailData);
}Thank you, and I hope this article helped you in some way. If you like the post, please support the article by sharing it. Have a nice day!👋
I am an mobile enthusiast 📱. My expertise is in mobile development, with more than 6 years of experience in the industry.
I would love to write technical blogs for you. Writing blogs and helping people is what I crave for 😊 .You can contact me at my email(abdulrehman0796@gmail.com) for any collaboration 👋
🔗 LinkedIn: https://www.linkedin.com/in/abdul-rehman-khilji/
