Klepnutím lze upravit styl předlohy. nadpisů. Kamil Říha Faculty of Electrical Engineering and Communication Department of Telecommunications

Signal Processing Laboratory

1 /18

SPLAB, BUT

Klepnutím lze upravit styl předlohy nadpisů.

http://spl.utko.feec.vutbr.cz

Kamil Říha Faculty of Electrical Engineering and Communication Department of Telecommunications

Czech Republic European Union


2 /52 /18

SPLAB, BUT

Group of Biomedical Signal Klepnutím lze upravit stylProcessing předlohy nadpisů. • Biomedical sensors design and signal processing • Signal processing for computer tomography and magnetic resonance 3D data • Ultrasound images and video sequences processing • Medical hardware designing (specialized microprocessor controlled equipment) • Diagnosis of nervous system disorders from speech

Brno University of Technology, Czech Republic


SPLAB, BUT

3 /52 /18

Klepnutím lzeOverview upravit styl předlohy nadpisů. common carotid artery (CCA ) in ultrasound scans

• • Ultrasound B-mode image processing – Preprocessing (de-noising, stabilization) – Tracking objects in a sequence – Data post-processing (analysis)

• Artery localization – in longitudinal scan – in transversal (perpendicular) scan



•

SPLAB, BUT

4 /52 /18

The common carotid artery Klepnutím lze upravit styl (CCA) předlohy nadpisů. Common carotid artery (carotis communis) is an artery that

supplies the head and neck with oxygenated blood • CCA ascends within the neck and it bifurcates into two branches – External carotid artery (ECA) • Supplying the exterior of the head and face

– Internal carotid artery (ICA) • Supplying the parts within the cranial and orbital cavities

Magnetic Resonance Angiography

Arteries of the neck Brno University of Technology, Czech Republic


SPLAB, BUT

5 /52 /18

The common carotid styl artery (CCA) Klepnutím lze upravit předlohy nadpisů. • Source of important information – Doctors can use it to evaluate the patients health

• Can be measured noninvasively by using ultrasound imaging – Superficial imaging – linear array transducer – Transverse x longitudinal sections linear array transducer

Transverse section

CCA

Longitudinal section Brno University of Technology, Czech Republic


SPLAB, BUT

6 /52 /18

Processing Klepnutím lze upravitChain styl předlohy nadpisů. A. Pre-processing

B. Processing

C. Post-processing

• Noise reduction – Noise in US images – Noise reduction with adaptive filters – Noise reduction with wavelet transformation

• Motion compensation – original method based on Optical flow analysis and vector field processing (pulsative movement suppression)



SPLAB, BUT

7 /52 /18

Motion compensation Klepnutím lze upravit styl předlohy nadpisů. • Why? – Doctors want to examine a certain region in video-sequence – This region is moving in recorded video-sequence • Because of “free-hand“ examination – probe movement

• Unwanted translational movement in sonographic videosequences

• Effort to eliminate such translation movement • Stabilization of examined region = motion compensation 7 Brno University of Technology, Czech Republic


8 /52 /18

SPLAB, BUT

Free-hand scanning Klepnutím lze upravit styl předlohy nadpisů. Main idea: translate image in inverse direction against the unwanted movement

Before motion compensation

After motion compensation 8 Brno University of Technology, Czech Republic


SPLAB, BUT

9 /52 /18

B. Medical image processing Klepnutím lze upravit styl předlohy nadpisů. A. Pre-processing

B. Processing

C. Post-processing

• Artery section area tracking in a video-sequence

9 Brno University of Technology, Czech Republic


10 /52 /18

SPLAB, BUT

Artery section areastyl tracking Klepnutím lze upravit předlohy nadpisů. Main goal: to develop a method for the tracking of an artery section area in its perpendicular or longitudinal cut – in the ultrasound scan. Typical perpendicular cut of an artery in sonographic image: The area inside the artery depends on blood pressure level. Systolic pressure level = maximal, diastolic pressure level = minimal artery section area. Videosequence contains the cardiac cycle data.



SPLAB, BUT

11 /52 /18

Good features to track Klepnutím lzedenoting upravit styl předlohy

nadpisů. The main goal of this stage is to denote the features being at the artery edge. Firstly, the Hessian is computed for every pixel in the area of interest:

𝐻 𝑓 𝑥, 𝑦

=

𝜕2𝑓 𝜕𝑥 2 𝜕2𝑓 𝜕𝑦𝑥

𝜕2𝑓 𝜕𝑥𝑦 𝜕2𝑓 𝜕𝑦 2

then some neighbourhood S(p) of every coordinate in resulting second derivative functions is counted in: 2 2

𝑀 𝑥, 𝑦 =

𝑆 𝑝

𝑆 𝑝

𝜕 𝑓 𝜕𝑥 2

𝜕2𝑓 𝜕𝑦𝑥

𝑆 𝑝

𝑆 𝑝

𝜕 𝑓 𝜕𝑥𝑦 𝜕2𝑓 𝜕𝑦 2

finally the “good feature” is detected based on eigenvalues of the matrix M. 11 Brno University of Technology, Czech Republic


SPLAB, BUT

12 /52 /18


nadpisů.

In practice…. initial ring defined by hand



13 /52 /18

SPLAB, BUT


nadpisů.

In practise…. detected features



SPLAB, BUT

14 /52 /18

Optical Flow – tracking procedure Klepnutím lze upravit styl předlohy nadpisů. Optical flow determination by Lucas-Kanade method: 𝐸x 𝑢 + 𝐸y 𝑣 + 𝐸t = 0 where 𝐸 𝑥, 𝑦, 𝑡 is the video sequence (or more likely the coordinates of features) and

𝐸x =

𝜕𝐸 𝜕𝑥

, 𝐸y =

𝜕𝐸 𝜕𝑦

, 𝐸t =

𝜕𝐸 𝜕𝑡

and 𝑢 =

𝑑𝑥 𝑑𝑡

,𝑣=

𝑑𝑦 𝑑𝑡

then the over determined equation 𝐸x1 𝐸x2 ⋮ 𝐸x𝑛

𝐸y1 𝐸y2 ⋮ 𝐸y𝑛

𝑢 𝑣

𝐸t1 𝐸 = − t2 ⋮ 𝐸t𝑛

is solved in the least squares sense



15 /52 /18

SPLAB, BUT

Optical Flow: tracking Klepnutím lze upravit stylprocedure předlohy Solving of this equationnadpisů. gives us the motion vector (u,v) , which implicitly means the coordinate of the feature in next frame and consequently in the whole videosequence. Now we have to compute the area of the artery as an ellipse fitted to feature points (in the sense of least square method) in every frame: The area of the ellipse is a searched value (in dependency on time) computed simply as:

𝑒𝑙𝑙𝑖𝑝𝑠𝑒 𝑎𝑟𝑒𝑎 = 𝜋 ∙ 𝑚𝑎𝑗𝑜𝑟 𝑎𝑥𝑖𝑠 ∙ 𝑚𝑖𝑛𝑜𝑟 𝑎𝑥𝑖𝑠



16 /52 /18

SPLAB, BUT

experimental Klepnutím lze upravitresults styl předlohy nadpisů. Final output: cardiac cycle curve the inner part of artery cross-section, expressed in terms of the number of pixels located within the artery circle, in dependence on time. The time is derived from the FPS of a given video-sequence (32 FPS).



17 /52 /18

SPLAB, BUT

Experimental Klepnutím lze upravitresults styl předlohy nadpisů.

….demonstration….



SPLAB, BUT

18 /52 /18

Experimental Klepnutím lze upravitresults styl předlohy nadpisů. • Different methods of analysation are possible (based on principle of fetures tracking) – Point-to-Point measurement • Very simple distance measurement

– Multipoint-to-multipoint measurement • More precise distance measurement

– Measurement of a distance between two splines • Measure a distance in many points between two curves perpendicularly to axis

– General shape measurement • Analyze general shape and measure inner size perpendicularly to axis

• These methods are implemented in application called sonoskop • Including the circular artery section area analysis, all the methods can be used for medical examination • The methods are not strictly for examination of arteries – They can be also used for heart examination, universal methods in industry etc. 18 Brno University of Technology, Czech Republic


19 /52 /18

SPLAB, BUT

Multipoint-to-Multipoint Klepnutímmeasurement lze upravit styl předlohy nadpisů.



20 /52 /18

SPLAB, BUT

Measurement of a Distance Between Klepnutím lze upravit styl předlohy Two Splines

nadpisů.



21 /52 /18

SPLAB, BUT

General lze shape measurement Klepnutím upravit styl předlohy nadpisů.



SPLAB, BUT

22 /52 /18

KlepnutímPost-processing lze upravit styl předlohy nadpisů. A. Pre-processing

B. Processing

C. Post-processing

• Normalization of different output curves produced by methods with different initialisation step (processing the same video sequence). • There is also a possibility to process the curves as a standard 1D signal (e.g. 1D median filtering). 22 Brno University of Technology, Czech Republic


SPLAB, BUT

23 /52 /18

Cardiaclze cycle computation Klepnutím upravit styl předlohy nadpisů. significant features for are fitted by ellipse and tracked through the video sequence Automatic initialization:

Hand initialization:



SPLAB, BUT

24 /52 /18

Cardiaclze cycle computation Klepnutím upravit styl předlohy nadpisů. the cardiac cycles (ellipse areas in dependence on time) are obtained by each method S [pxl2]

5000 4500

4000

𝑆1

3500

3000 2500

2000

𝑆2 𝑡

When we want to compare both results, we have to normalize these curves considering the acquisition physical principle.



SPLAB, BUT

25 /52 /18

KlepnutímPostprocessing lze upravit styl předlohy nadpisů. after the DC component removal S - Savg [pxl2]

500

𝑆avg = 300

1 𝑛

𝑛

𝑠𝑖 𝑖=1

100

-100

𝑡

-300

-500 25 Brno University of Technology, Czech Republic


SPLAB, BUT

26 /52 /18

KlepnutímPostprocessing lze upravit styl předlohy nadpisů. after the window-based DC component removal

∆S [pxl2]

400

there are differences because absolute values of 𝑆2 (gray) are smaller than corresponding values of 𝑆1 (black)

𝑆avg_w 𝑖 =

1 𝑤

𝑤 2

𝑠𝑖+𝑤 𝑤 𝑖=− 2

200 0

𝑡

-200 -400

∆𝑆 𝑖 = 𝑠𝑖 − 𝑆avg_w 𝑖 26 Brno University of Technology, Czech Republic


SPLAB, BUT

27 /52 /18

KlepnutímPostprocessing lze upravit styl předlohy nadpisů. the movement of points localised on the artery border is independent on the radius of detected area - the radius change is constant, independent on the circle size area of i-th circle: ∆

∆

𝑠𝑖 = 𝜋𝑟 2 𝑖 radius of i-th circle:

𝑟 𝑖 =

∆

2

∆

1

∆𝑟 𝑖 =

𝑠𝑖 𝜋 𝑆avg_w 𝑠𝑖 − 𝜋 𝜋



SPLAB, BUT

28 /52 /18

KlepnutímPostprocessing lze upravit styl předlohy nadpisů. after the calculation of radius change ∆𝑟 𝑖 =

𝑆avg_w 𝑠𝑖 − 𝜋 𝜋

∆r [pxl]

1,5 1 0,5 0 -0,5 -1 -1,5

t [sec] 28 Brno University of Technology, Czech Republic


SPLAB, BUT

29 /52 /18

Partial Conclusion Klepnutím lze upravit styl předlohy nadpisů. A. Pre-processing

B. Processing

C. Post-processing

• Basic chain for ultrasound medical video sequences processing methods and results have been presented • Mainly for carotis communis analysis • Automatic detection is needed for the method initialisation



30 /52 /18

SPLAB, BUT

Artery localization in longitudinal scan Klepnutím lze upravit styl předlohy nadpisů. Input: image of longitudinal section of CCA Output: localized CCA in image



•

SPLAB, BUT

31 /52 /18

PROBLEMS KlepnutímDETECTION lze upravit styl předlohy Artery localization nadpisů. – State of the art in artery localization (longitudinal section, US) • Manual • Automatic (not robust – problems with non-horizontally arteries, curved arteries)

non-horizontally arteries

curved arteries 31 Brno University of Technology, Czech Republic


SPLAB, BUT

32 /52 /18

PROPOSED METHOD FOR ARTERY Klepnutím lze upravit stylLOCALIZATION předlohy nadpisů. Classification Sampling, feature extraction

SVM classifier

px

features

[1,1]

[1.0,5.3,3.1,…]

[5,2]

[1.1,3.3,1.1,…]

[7,2]

[0.0,2.3,7.8,…]

[10,1]

[2.5,5.85,2.1,…]

…

…

Post-processing Artery points selection

RANSAC method



SPLAB, BUT

33 /52 /18

SAMPLING Klepnutím lze, FEATURE upravitEXTRACTION styl předlohy nadpisů. Classification Sampling, feature extraction

SVM classifier


RANSAC method

• Sampling Point 1

Point 2

Point 3

sampling Point 4



SPLAB, BUT

34 /52 /18

Extraction lze of local image Klepnutím upravit styl features předlohy nadpisů. • Subsequent classification is based on local image features

– Local image features = parameters obtained form the pixels and their neighborhood

• Examples of local image features – – – – – –

Mean value Standard deviation Max, median, min values Derivatives (Sobel, Laplacian) Haar features etc. Examples of rectangular neighborhoods 34 Brno University of Technology, Czech Republic


SPLAB, BUT

35 /52 /18

Extractionlze of upravit local image Klepnutím stylfeatures předlohy nadpisů. • Local features – computed in certain neighborhood (rectangular, circular, etc.) of sampled pixels and formatted into the table Sampled input image Point 1

Point 2

Point 3

Point 4

Local features for particular points

compute features

Feature 1

Feature 2

…

Point 1

10.851

128.553

…

Point 2

11.524

155.984

…

Point 3

9.881

105.632

…

Point 4

13.235

156.667

…

…

…

…

…

e.g. mean value e.g. standard deviation



SPLAB, BUT

36 /52 /18

SVMupravit CLASSIFIER Klepnutím lze styl předlohy nadpisů. Classification Sampling, feature extraction

SVM classifier


RANSAC method

• Classification is performed on the basis of local features • Two class classifying of pixels – “artery pixels” (depicted as white pixels) – “other pixels” (depicted as black pixels)

• SVM classifier – Support vector machine classification – We test two implementations • Classical implementation • Extended implementation (radial basis function mapping) 36 Brno University of Technology, Czech Republic


SPLAB, BUT

37 /52 /18

SELECTION OF POINTS Klepnutím lze upravit styl předlohy nadpisů. Classification Sampling, feature extraction

•

SVM classifier


RANSAC method

In further processing only pixels classified as “artery pixels” are considered

You can see, that there are some misclassified pixels

• Why there are misclassified points? – Similarity of tissue, poor accuracy of classifier, bad choice of features 37 Brno University of Technology, Czech Republic


SPLAB, BUT

38 /52 /18

RANSAC ALGORITHM Klepnutím lze upravit styl předlohy nadpisů. Classification Sampling, feature extraction

•

SVM classifier


RANSAC method

RANSAC algorithm – suppresses points that were misclassified as “artery pixels” – Misclassified points form small clusters, or they are isolated points

–

RANSAC algorithm – find the best linear model of artery



SPLAB, BUT

39 /52 /18

PROPOSED METHOD FOR ARTERY LOCALIZATION Klepnutím lze upravit styl předlohy IMPROVEMENT FOR CURVED ARTERIES

nadpisů. Classification Sampling, feature extraction


SVM classifier

px

features

[1,1]

[1.0,5.3,3.1,…]

[5,2]

[1.1,3.3,1.1,…]

[7,2]

[0.0,2.3,7.8,…]

[10,1]

[2.5,5.85,2.1,…]

…

…

px

features

label

[1,1]

[1.0,5.3,3.1,…]

0

[5,2]

[1.1,3.3,1.1,…]

0

[7,2]

[0.0,2.3,7.8,…]

1

[10,1]

[2.5,5.85,2.1,…]

1

…

…

Proposed RANSAC based method



SPLAB, BUT

40 /52 /18

LASSIFICATION KlepnutímClze upravit STEP styl předlohy nadpisů. Classification Sampling, feature extraction

SVM classifier



• Classification + Artery points selection – SVM (Support Vector Machine) classifier • Two category classification (Artery pixels X Other pixels)

Highlighted artery pixels

• Feature set for classifier - local image features

– In further processing only “Artery pixels” • Select only pixels classified as “Artery pixels”

•

Due to the classifier inaccuracy – misclassified pixels –

Post-processing 40 Brno University of Technology, Czech Republic


SPLAB, BUT

41 /52 /18

STEP předlohy KlepnutímPOST lze-PROCESSING upravit styl nadpisů. Classification Sampling, feature extraction

SVM classifier



• Suppress misclassified pixels

– Problem of data containing outliers  RANSAC algorithm 41 Brno University of Technology, Czech Republic


SPLAB, BUT

42 /52 /18

PROPOSEDlze RANSAC-B Klepnutím upravitASED stylMETHOD předlohy nadpisů. Classification Sampling, feature extraction

SVM classifier



• RANSAC • Standard linear model is inapplicable – artery can be curved • Proposed modification of standard RANSAC algorithm – Non-linear mathematical model - explicit polynomial curve (second order) – 𝑦 = 𝑝2 𝑥 2 + 𝑝1 𝑥 + 𝑝0 .



SPLAB, BUT

43 /52 /18

Artery detection in transversal scan Klepnutím lze upravit styl předlohy • Detection of artery innadpisů. ultrasound image of perpendicular cut – The area inside is correlated with blood pressure level • Artery section area – can be modelled with ellipse shape

– Videosequence contains the cardiac cycle data • Systolic pressure level = maximal artery section area • diastolic pressure level = minimal artery section area

Sample ultrasound image

Artery section 43 Brno University of Technology, Czech Republic


SPLAB, BUT

44 /52 /18

Artery detection in transversal scan Klepnutím lze upravit styl předlohy

nadpisů. Circular pattern localization

•

The aim: •

•

•

Optical flow estimation Average image calculation Median filtration, thresholding Hough transform

The aim: •

Detection of artery

Methods: • • • •

Exact radius determination

•

Precise localization

Methods: •

Analysis of average intensity inside the ring



45 /52 /18

SPLAB, BUT

Circularlze pattern localization Klepnutím upravit styl předlohy nadpisů. 1) Input: source frames in video-sequence



SPLAB, BUT

46 /52 /18

Circularlze pattern localization Klepnutím upravit styl předlohy nadpisů. 2) Scalar field of average optical flow (average value of the movement) in the video sequence



47 /52 /18

SPLAB, BUT

Circularlze pattern localization Klepnutím upravit styl předlohy nadpisů. 3) After the dilation step



48 /52 /18

SPLAB, BUT

Circularlze pattern localization Klepnutím upravit styl předlohy nadpisů. 4) After thresholding



49 /52 /18

SPLAB, BUT

Circularlze pattern localization Klepnutím upravit styl předlohy nadpisů. 5) After the median filtering step



50 /52 /18

SPLAB, BUT

Circularlze pattern localization Klepnutím upravit styl předlohy nadpisů. 6) Hough Transform: detected circle (position of artery)



SPLAB, BUT

51 /52 /18

Circularlze pattern localization Klepnutím upravit styl předlohy nadpisů. Detected circle – maximum in accumulation array (Hough space) Multiple circles – more peaks in accumulator

• Bayes classifier is used to choose the most probable circle – The feature vector for the classifier consists of statistic values acquired from the pixel intensity localised inside the manually denoted circles from the training data set (average value and standard deviation) and for a circle of smaller radius and a circle from binary image



SPLAB, BUT

52 /52 /18

Future work Klepnutím lze upravit styl předlohy nadpisů.

• real time and long-time tracking (problems with re-initialization of the tracked object) • Using of genetic programming approach for the higher reliability of the algorithm • Automatic detection and tracking of more complex objects (heard ventricle)



53 /18

SPLAB, BUT

Klepnutím lze upravit styl předlohy nadpisů.

http://spl.utko.feec.vutbr.cz

Acknowledgement: This work was also prepared with the support of the MSMT project no. 2B06111: The Research of Algorithms for Processing of Digital Images and Image Sequences and the MSMT project The Research of Algorithms for Processing of Digital Images and Image Sequences, code: ME10123. Some functions from the OpenCV library (Open Source Computer Vision) were used for the implementation of algorithms. Czech Republic European Union

Klepnutím lze upravit styl předlohy. nadpisů. Kamil Říha Faculty of Electrical Engineering and Communication Department of Telecommunications

Recommend Documents